[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:40:46 dist-test-slave-zr1t krb5kdc[23967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:40:46 dist-test-slave-zr1t krb5kdc[23967](info): set up 2 sockets
Aug 06 22:40:46 dist-test-slave-zr1t krb5kdc[23967](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 22:40:47.854383 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:40:47 dist-test-slave-zr1t krb5kdc[23967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:40:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:40:48.063208 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:44795
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:44795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:48.401247 23984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:48.401665 23984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:48.401857 23984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:48.409461 23984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:40:48.409685 23984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:48.409773 23984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:48.409870 23984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:40:48.409940 23984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:40:48.420152 23984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:44795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:44795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:48.421578 23984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:48.422953 23984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:48.435480 23994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:48.435446 23992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:48.435520 23990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:48.436306 23984 server_base.cc:1034] running on GCE node
I20260806 22:40:48.437103 23984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:48.438666 23984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:48.439908 23984 hybrid_clock.cc:648] HybridClock initialized: now 1786056048439865 us; error 63 us; skew 500 ppm
I20260806 22:40:48.440279 23984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:48 dist-test-slave-zr1t krb5kdc[23967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056048, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:48.450874 23984 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:40:48.452596 23984 webserver.cc:460] Webserver started at http://127.23.100.190:40921/ using document root <none> and password file <none>
I20260806 22:40:48.453328 23984 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:48.453482 23984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:48.453833 23984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:48.457871 23984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "382d734a086a41c68d78916efd2c5a16"
format_stamp: "Formatted at 2026-08-06 22:40:48 on dist-test-slave-zr1t"
server_key: "a8c4b41076cf84f5e4ec8a8a06480f79"
server_key_iv: "06e7eb54875fd9557dd8af431eb929f8"
server_key_version: "encryptionkey@0"
I20260806 22:40:48.458683 23984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "382d734a086a41c68d78916efd2c5a16"
format_stamp: "Formatted at 2026-08-06 22:40:48 on dist-test-slave-zr1t"
server_key: "a8c4b41076cf84f5e4ec8a8a06480f79"
server_key_iv: "06e7eb54875fd9557dd8af431eb929f8"
server_key_version: "encryptionkey@0"
I20260806 22:40:48.464401 23984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 22:40:48.468231 24004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:48.470011 23984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:40:48.470249 23984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "382d734a086a41c68d78916efd2c5a16"
format_stamp: "Formatted at 2026-08-06 22:40:48 on dist-test-slave-zr1t"
server_key: "a8c4b41076cf84f5e4ec8a8a06480f79"
server_key_iv: "06e7eb54875fd9557dd8af431eb929f8"
server_key_version: "encryptionkey@0"
I20260806 22:40:48.470505 23984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:48.494517 23984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:48.499524 23984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:48.499971 23984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:48.519974 23984 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:44795
I20260806 22:40:48.520006 24074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:44795 every 8 connection(s)
I20260806 22:40:48.521431 23984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:40:48.525070 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 23984
I20260806 22:40:48.525434 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:40:48.526000 24075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:48.528162 23954 external_mini_cluster.cc:1428] Setting key 82ee9e3a5ce5aedfcec6a0a02c622553
I20260806 22:40:48.538497 24075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16: Bootstrap starting.
I20260806 22:40:48.541488 24075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:48.542728 24075 log.cc:824] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16: Log is configured to *not* fsync() on all Append() calls
I20260806 22:40:48.546448 24075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16: No bootstrap required, opened a new log
I20260806 22:40:48.555428 24075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382d734a086a41c68d78916efd2c5a16" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44795 } }
I20260806 22:40:48.555814 24075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:48.555953 24075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382d734a086a41c68d78916efd2c5a16, State: Initialized, Role: FOLLOWER
I20260806 22:40:48.556556 24075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382d734a086a41c68d78916efd2c5a16" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44795 } }
I20260806 22:40:48.556787 24075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:40:48.556941 24075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:40:48.557111 24075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 22:40:48 dist-test-slave-zr1t krb5kdc[23967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:48.560246 24075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382d734a086a41c68d78916efd2c5a16" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44795 } }
I20260806 22:40:48.560874 24075 leader_election.cc:302] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 382d734a086a41c68d78916efd2c5a16; no voters:
I20260806 22:40:48.561502 24075 leader_election.cc:288] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:40:48.561704 24081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:40:48.564000 24075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:40:48.568645 24081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [term 1 LEADER]: Becoming Leader. State: Replica: 382d734a086a41c68d78916efd2c5a16, State: Running, Role: LEADER
I20260806 22:40:48.569202 24081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382d734a086a41c68d78916efd2c5a16" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44795 } }
I20260806 22:40:48.572506 24082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "382d734a086a41c68d78916efd2c5a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382d734a086a41c68d78916efd2c5a16" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44795 } } }
I20260806 22:40:48.572567 24083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 382d734a086a41c68d78916efd2c5a16. Latest consensus state: current_term: 1 leader_uuid: "382d734a086a41c68d78916efd2c5a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382d734a086a41c68d78916efd2c5a16" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44795 } } }
I20260806 22:40:48.572983 24082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [sys.catalog]: This master's current role is: LEADER
I20260806 22:40:48.573009 24083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16 [sys.catalog]: This master's current role is: LEADER
I20260806 22:40:48.573925 24091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:40:48.575938 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:48.532929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.23.100.190:44795)
0806 22:40:48.533927 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:48.533942 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:48.535298 (+ 1356us) server_negotiation.cc:407] Connection header received
0806 22:40:48.544846 (+ 9548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:48.544872 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:48.545363 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:48.545823 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:48.548607 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:48.550694 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:48.552354 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:48.553440 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:48.553709 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:48.560579 (+ 6870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:48.560589 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:48.560605 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:48.560671 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:48.566356 (+ 5685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:48.567870 (+ 1514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:48.567883 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:48.567895 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:48.568192 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:48.569003 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:48.569012 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:48.569019 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:48.570104 (+ 1085us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:48.570588 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:48.573758 (+ 3170us) server_negotiation.cc:299] Negotiation successful
0806 22:40:48.574240 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":472,"threads_started":1}
I20260806 22:40:48.583046 24091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:40:48.594054 24091 catalog_manager.cc:1347] Generated new cluster ID: 3be749818310422cabf3f96ea2104a91
I20260806 22:40:48.594195 24091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:40:48.613974 24091 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:40:48.615319 24091 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:40:48.622722 24091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 382d734a086a41c68d78916efd2c5a16: Generated new TSK 0
I20260806 22:40:48.623728 24091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:40:48.682868 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44795
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:49.029043 24106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:49.029345 24106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:49.029449 24106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:49.037072 24106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:49.037192 24106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:49.037317 24106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:40:49.047560 24106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:49.049275 24106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:49.051086 24106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:49.065042 24111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:49.065012 24112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:49.064972 24116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:49.066241 24106 server_base.cc:1034] running on GCE node
I20260806 22:40:49.066874 24106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:49.067873 24106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:49.069108 24106 hybrid_clock.cc:648] HybridClock initialized: now 1786056049069052 us; error 52 us; skew 500 ppm
I20260806 22:40:49.069438 24106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:49 dist-test-slave-zr1t krb5kdc[23967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056049, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:49.072685 24106 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:40:49.074666 24106 webserver.cc:460] Webserver started at http://127.23.100.129:38471/ using document root <none> and password file <none>
I20260806 22:40:49.075482 24106 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:49.075635 24106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:49.075968 24106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:49.078958 24106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "42ad0223ffb748969c3f9a81577e550f"
format_stamp: "Formatted at 2026-08-06 22:40:49 on dist-test-slave-zr1t"
server_key: "d5101c43f230a16b39af7c3ce9f56fa7"
server_key_iv: "b95daa1aa9d58a7ef647f3ea5de6595a"
server_key_version: "encryptionkey@0"
I20260806 22:40:49.079737 24106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "42ad0223ffb748969c3f9a81577e550f"
format_stamp: "Formatted at 2026-08-06 22:40:49 on dist-test-slave-zr1t"
server_key: "d5101c43f230a16b39af7c3ce9f56fa7"
server_key_iv: "b95daa1aa9d58a7ef647f3ea5de6595a"
server_key_version: "encryptionkey@0"
I20260806 22:40:49.085062 24106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:40:49.088634 24125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:49.090024 24106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:40:49.090209 24106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "42ad0223ffb748969c3f9a81577e550f"
format_stamp: "Formatted at 2026-08-06 22:40:49 on dist-test-slave-zr1t"
server_key: "d5101c43f230a16b39af7c3ce9f56fa7"
server_key_iv: "b95daa1aa9d58a7ef647f3ea5de6595a"
server_key_version: "encryptionkey@0"
I20260806 22:40:49.090438 24106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:49.114090 24106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:49.118497 24106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:49.118907 24106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:49.119956 24106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:40:49.121477 24106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:40:49.121601 24106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 22:40:49.121758 24106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:40:49.121833 24106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:49.156745 24106 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:34073
I20260806 22:40:49.156821 24278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:34073 every 8 connection(s)
I20260806 22:40:49.158135 24106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:40:49.165984 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24106
I20260806 22:40:49.166267 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:40:49.167088 23954 external_mini_cluster.cc:1428] Setting key ff3a3669d81a8b4113855616c3df458d
Aug 06 22:40:49 dist-test-slave-zr1t krb5kdc[23967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056049, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:49.181067 24283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:49.160813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:37647 (local address 127.23.100.190:44795)
0806 22:40:49.161265 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:49.161270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:49.162329 (+ 1059us) server_negotiation.cc:407] Connection header received
0806 22:40:49.164188 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:49.164192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:49.164278 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:49.164374 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:49.166119 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:49.167271 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:49.169095 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:49.169703 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:49.169844 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:49.174117 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:49.174127 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:49.174134 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:49.174183 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:49.177426 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:49.178303 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:49.178312 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:49.178318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:49.178491 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:49.178929 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:49.178937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:49.178943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:49.179366 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:49.179533 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:49.180297 (+ 764us) server_negotiation.cc:299] Negotiation successful
0806 22:40:49.180580 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260806 22:40:49.181890 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:49.161367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:44795 (local address 127.23.100.129:37647)
0806 22:40:49.162145 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:49.162183 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:49.163912 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:49.164662 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:49.164670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:49.165209 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:49.165927 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:49.165945 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:49.167483 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:49.167487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:49.168942 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:49.168960 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:49.169947 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:49.169952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:49.170158 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:49.171043 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:49.171067 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:49.173856 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:49.177602 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:49.177612 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:49.177637 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:49.178149 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:49.178610 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:49.178613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:49.178616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:49.178807 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:49.179506 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:49.179513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:49.179851 (+ 338us) client_negotiation.cc:769] Sending connection context
0806 22:40:49.180124 (+ 273us) client_negotiation.cc:240] Negotiation successful
0806 22:40:49.180544 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":328,"threads_started":1}
I20260806 22:40:49.183815 24281 heartbeater.cc:347] Connected to a master server at 127.23.100.190:44795
I20260806 22:40:49.184266 24281 heartbeater.cc:464] Registering TS with master...
I20260806 22:40:49.185212 24281 heartbeater.cc:511] Master 127.23.100.190:44795 requested a full tablet report, sending...
I20260806 22:40:49.187849 24029 ts_manager.cc:194] Registered new tserver with Master: 42ad0223ffb748969c3f9a81577e550f (127.23.100.129:34073)
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
I20260806 22:40:49.190001 24029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:37647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:40:49.214896 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44795
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:49.544575 24289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:49.544840 24289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:49.544938 24289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:49.552018 24289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:49.552131 24289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:49.552304 24289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:40:49.561843 24289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:49.563154 24289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:49.564533 24289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:49.577987 24297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:49.578038 24298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:49.578140 24300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:49.578768 24289 server_base.cc:1034] running on GCE node
I20260806 22:40:49.579362 24289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:49.580271 24289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:49.581504 24289 hybrid_clock.cc:648] HybridClock initialized: now 1786056049581468 us; error 53 us; skew 500 ppm
I20260806 22:40:49.581933 24289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:49 dist-test-slave-zr1t krb5kdc[23967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056049, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:49.585083 24289 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:40:49.586587 24289 webserver.cc:460] Webserver started at http://127.23.100.130:45739/ using document root <none> and password file <none>
I20260806 22:40:49.587242 24289 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:49.587345 24289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:49.587608 24289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:49.590716 24289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "02b8d32a91da4c1cad17602c2788f553"
format_stamp: "Formatted at 2026-08-06 22:40:49 on dist-test-slave-zr1t"
server_key: "d78712f36ce4f3494e68ef00038ac935"
server_key_iv: "3676591d5af2ff688e3bb289c90406eb"
server_key_version: "encryptionkey@0"
I20260806 22:40:49.591454 24289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "02b8d32a91da4c1cad17602c2788f553"
format_stamp: "Formatted at 2026-08-06 22:40:49 on dist-test-slave-zr1t"
server_key: "d78712f36ce4f3494e68ef00038ac935"
server_key_iv: "3676591d5af2ff688e3bb289c90406eb"
server_key_version: "encryptionkey@0"
I20260806 22:40:49.596822 24289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 22:40:49.600374 24312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:49.601753 24289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:40:49.601886 24289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "02b8d32a91da4c1cad17602c2788f553"
format_stamp: "Formatted at 2026-08-06 22:40:49 on dist-test-slave-zr1t"
server_key: "d78712f36ce4f3494e68ef00038ac935"
server_key_iv: "3676591d5af2ff688e3bb289c90406eb"
server_key_version: "encryptionkey@0"
I20260806 22:40:49.602056 24289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:49.619689 24289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:49.624248 24289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:49.624614 24289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:49.626026 24289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:40:49.627801 24289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:40:49.627909 24289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:49.628041 24289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:40:49.628119 24289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:49.663123 24289 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:44503
I20260806 22:40:49.663182 24469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:44503 every 8 connection(s)
I20260806 22:40:49.664587 24289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:40:49.674676 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24289
I20260806 22:40:49.674890 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:40:49.675365 23954 external_mini_cluster.cc:1428] Setting key fdad38d946ced9636442c52a29a0e31f
Aug 06 22:40:49 dist-test-slave-zr1t krb5kdc[23967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056049, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:49.685309 24283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:49.667335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:53861 (local address 127.23.100.190:44795)
0806 22:40:49.667528 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:49.667533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:49.668751 (+ 1218us) server_negotiation.cc:407] Connection header received
0806 22:40:49.670622 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:49.670628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:49.670713 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:49.670837 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:49.672292 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:49.673016 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:49.674880 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:49.675482 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:49.675652 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:49.679437 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:49.679442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:49.679446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:49.679479 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:49.682162 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:49.683032 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:49.683037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:49.683043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:49.683171 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:49.683614 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:49.683619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:49.683624 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:49.683893 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:49.684047 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:49.684726 (+ 679us) server_negotiation.cc:299] Negotiation successful
0806 22:40:49.684956 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 22:40:49.686362 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:49.667853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:44795 (local address 127.23.100.130:53861)
0806 22:40:49.668581 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:49.668617 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:49.670351 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:49.671077 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:49.671084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:49.671577 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:49.672116 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:49.672132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:49.673188 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:49.673193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:49.674735 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:49.674752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:49.675803 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:49.675811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:49.676089 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:49.676792 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:49.676819 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:49.679199 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:49.682321 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:49.682333 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:49.682350 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:49.682897 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:49.683288 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:49.683293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:49.683297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:49.683490 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:49.684025 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:49.684033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:49.684361 (+ 328us) client_negotiation.cc:769] Sending connection context
0806 22:40:49.684625 (+ 264us) client_negotiation.cc:240] Negotiation successful
0806 22:40:49.685146 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":297,"threads_started":1}
I20260806 22:40:49.688906 24470 heartbeater.cc:347] Connected to a master server at 127.23.100.190:44795
I20260806 22:40:49.689220 24470 heartbeater.cc:464] Registering TS with master...
I20260806 22:40:49.690017 24470 heartbeater.cc:511] Master 127.23.100.190:44795 requested a full tablet report, sending...
I20260806 22:40:49.691601 24029 ts_manager.cc:194] Registered new tserver with Master: 02b8d32a91da4c1cad17602c2788f553 (127.23.100.130:44503)
I20260806 22:40:49.692875 24029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:53861
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:40:49.720608 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44795
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:50.051441 24475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:50.051738 24475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:50.051910 24475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:50.059618 24475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:50.059737 24475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:50.059948 24475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:40:50.070034 24475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:50.071331 24475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:50.072698 24475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:50.086150 24483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:50.086104 24486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:50.086169 24484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:50.087339 24475 server_base.cc:1034] running on GCE node
I20260806 22:40:50.088012 24475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:50.088991 24475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:50.090226 24475 hybrid_clock.cc:648] HybridClock initialized: now 1786056050090172 us; error 67 us; skew 500 ppm
I20260806 22:40:50.090591 24475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:50 dist-test-slave-zr1t krb5kdc[23967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:50.093927 24475 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:40:50.095485 24475 webserver.cc:460] Webserver started at http://127.23.100.131:42965/ using document root <none> and password file <none>
I20260806 22:40:50.096174 24475 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:50.096326 24475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:50.096652 24475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:50.100109 24475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "e013d33ea6ef4fb8923dbac922ef9bf0"
format_stamp: "Formatted at 2026-08-06 22:40:50 on dist-test-slave-zr1t"
server_key: "cd0c82d749995cca9e3f4d199b3e6a17"
server_key_iv: "8a33bf0ddf5f09c43d6dbcfcd8405189"
server_key_version: "encryptionkey@0"
I20260806 22:40:50.100929 24475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "e013d33ea6ef4fb8923dbac922ef9bf0"
format_stamp: "Formatted at 2026-08-06 22:40:50 on dist-test-slave-zr1t"
server_key: "cd0c82d749995cca9e3f4d199b3e6a17"
server_key_iv: "8a33bf0ddf5f09c43d6dbcfcd8405189"
server_key_version: "encryptionkey@0"
I20260806 22:40:50.106304 24475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 22:40:50.110000 24494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:50.111274 24475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:40:50.111451 24475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "e013d33ea6ef4fb8923dbac922ef9bf0"
format_stamp: "Formatted at 2026-08-06 22:40:50 on dist-test-slave-zr1t"
server_key: "cd0c82d749995cca9e3f4d199b3e6a17"
server_key_iv: "8a33bf0ddf5f09c43d6dbcfcd8405189"
server_key_version: "encryptionkey@0"
I20260806 22:40:50.111660 24475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:50.151701 24475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:50.158449 24475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:50.158856 24475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:50.160190 24475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:40:50.162185 24475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:40:50.162364 24475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 22:40:50.162544 24475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:40:50.162655 24475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:50.193665 24281 heartbeater.cc:503] Master 127.23.100.190:44795 was elected leader, sending a full tablet report...
I20260806 22:40:50.206218 24475 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:38189
I20260806 22:40:50.206254 24658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:38189 every 8 connection(s)
I20260806 22:40:50.207639 24475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:40:50.210520 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24475
I20260806 22:40:50.210736 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:40:50.211275 23954 external_mini_cluster.cc:1428] Setting key e726a8fd63b376e0b4156733b114403d
Aug 06 22:40:50 dist-test-slave-zr1t krb5kdc[23967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:50.231911 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:50.210689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:46361 (local address 127.23.100.190:44795)
0806 22:40:50.211216 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:50.211223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:50.212533 (+ 1310us) server_negotiation.cc:407] Connection header received
0806 22:40:50.214979 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:50.214984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:50.215077 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:50.215193 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:50.216959 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.217849 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.219829 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.220218 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.220313 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:50.224147 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:50.224152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:50.224156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:50.224190 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:50.227843 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.228992 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.229000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.229006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.229162 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.229810 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.229815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.229818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.230095 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:50.230263 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:50.231294 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0806 22:40:50.231529 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":263,"threads_started":1}
I20260806 22:40:50.233548 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:50.211457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:44795 (local address 127.23.100.131:46361)
0806 22:40:50.212367 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:50.212418 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:50.214609 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:50.215448 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:50.215457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:50.216127 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:50.216783 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:50.216803 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.217999 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.218004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:50.219651 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:50.219679 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.220486 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.220497 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:50.220774 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:50.221459 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:50.221501 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:50.223931 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:50.228043 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:50.228056 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:50.228076 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:50.228820 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:50.229336 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:50.229344 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:50.229349 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:50.229660 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:50.230244 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:50.230252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:50.230744 (+ 492us) client_negotiation.cc:769] Sending connection context
0806 22:40:50.231161 (+ 417us) client_negotiation.cc:240] Negotiation successful
0806 22:40:50.231698 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":399,"threads_started":1}
I20260806 22:40:50.235201 24659 heartbeater.cc:347] Connected to a master server at 127.23.100.190:44795
I20260806 22:40:50.235574 24659 heartbeater.cc:464] Registering TS with master...
I20260806 22:40:50.236527 24659 heartbeater.cc:511] Master 127.23.100.190:44795 requested a full tablet report, sending...
I20260806 22:40:50.238097 24029 ts_manager.cc:194] Registered new tserver with Master: e013d33ea6ef4fb8923dbac922ef9bf0 (127.23.100.131:38189)
I20260806 22:40:50.239138 24029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:46361
I20260806 22:40:50.240176 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 22:40:50 dist-test-slave-zr1t krb5kdc[23967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 22:40:50 dist-test-slave-zr1t krb5kdc[23967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:40:50.305356 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:50.282563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.23.100.129:34073)
0806 22:40:50.284033 (+ 1470us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:50.284040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:50.284068 (+ 28us) server_negotiation.cc:407] Connection header received
0806 22:40:50.284140 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:50.284147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:50.284422 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:50.284630 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:50.286749 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.288501 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.290716 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.291685 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.292046 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:50.298291 (+ 6245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:50.298307 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:50.298323 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:50.298370 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:50.302325 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.303165 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.303173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.303175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.303317 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.303656 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.303660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.303662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.304016 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:50.304246 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:50.304692 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 22:40:50.305018 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":475,"threads_started":1}
W20260806 22:40:50.306933 24168 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40694
Aug 06 22:40:50 dist-test-slave-zr1t krb5kdc[23967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 22:40:50 dist-test-slave-zr1t krb5kdc[23967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:40:50.366303 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:50.345581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.23.100.129:34073)
0806 22:40:50.346524 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:50.346536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:50.346566 (+ 30us) server_negotiation.cc:407] Connection header received
0806 22:40:50.346651 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:50.346659 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:50.346808 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:50.346945 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:50.349238 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.350577 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.353090 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.354003 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.354143 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:50.360535 (+ 6392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:50.360541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:50.360544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:50.360587 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:50.363803 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.364589 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.364594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.364596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.364732 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.365084 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.365087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.365089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.365340 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:50.365599 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:50.365886 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 22:40:50.366070 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 22:40:50.367935 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24106
I20260806 22:40:50.380971 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:50.369026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.23.100.129:34073)
0806 22:40:50.369223 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:50.369230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:50.369594 (+ 364us) server_negotiation.cc:407] Connection header received
0806 22:40:50.369769 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:50.369773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:50.369829 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:50.369904 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:50.371171 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.371872 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.373166 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.373764 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.373857 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:50.376308 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:50.376312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:50.376316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:50.376344 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:50.378834 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.379439 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.379443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.379445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.379579 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.379883 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.379886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.379888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.380127 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:50.380301 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:50.380553 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 22:40:50.380706 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 22:40:50.381767 24274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:40:50.466271 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24106
I20260806 22:40:50.494971 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24289
Aug 06 22:40:50 dist-test-slave-zr1t krb5kdc[23967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:40:50.511714 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:50.495913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.23.100.130:44503)
0806 22:40:50.496533 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:50.496540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:50.496568 (+ 28us) server_negotiation.cc:407] Connection header received
0806 22:40:50.496641 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:50.496648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:50.496847 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:50.496990 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:50.498428 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.499570 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.500929 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.501495 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.501774 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:50.506024 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:50.506036 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:50.506051 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:50.506084 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:50.508984 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.509700 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.509708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.509710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.509841 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.510220 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.510223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.510225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.510560 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:50.510791 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:50.511075 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 22:40:50.511275 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":333,"threads_started":1}
I20260806 22:40:50.513036 24462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:40:50.595747 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24289
I20260806 22:40:50.621047 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24475
Aug 06 22:40:50 dist-test-slave-zr1t krb5kdc[23967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:40:50.640038 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:50.622276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.23.100.131:38189)
0806 22:40:50.622870 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:50.622877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:50.622910 (+ 33us) server_negotiation.cc:407] Connection header received
0806 22:40:50.622972 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:50.622979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:50.623189 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:50.623366 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:50.624708 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.625770 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.627072 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:50.627985 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:50.628121 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:50.633462 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:50.633476 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:50.633491 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:50.633641 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:50.637313 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.638072 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.638079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.638082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.638207 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:50.638552 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:50.638555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:50.638557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:50.638912 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:50.639157 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:50.639391 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 22:40:50.639604 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":268,"threads_started":1}
I20260806 22:40:50.641358 24652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:40:50.724944 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24475
I20260806 22:40:50.746135 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 23984
I20260806 22:40:50.747326 24069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:40:50.852272 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 23984
2026-08-06T22:40:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4335 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:40:50 dist-test-slave-zr1t krb5kdc[24687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:40:50 dist-test-slave-zr1t krb5kdc[24687](info): set up 2 sockets
Aug 06 22:40:50 dist-test-slave-zr1t krb5kdc[24687](info): commencing operation
krb5kdc: starting...
W20260806 22:40:52.143487 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:40:52 dist-test-slave-zr1t krb5kdc[24687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:40:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:40:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:40:52.352979 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:39395
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:39395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:52.688342 24705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:52.688726 24705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:52.688903 24705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:52.696221 24705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:40:52.696403 24705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:52.696521 24705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:52.696626 24705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:40:52.696717 24705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:40:52.706995 24705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:39395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:39395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:52.708455 24705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:52.709859 24705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:52.722467 24711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:52.722543 24710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:52.722581 24713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:52.724241 24705 server_base.cc:1034] running on GCE node
I20260806 22:40:52.725194 24705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:52.726872 24705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:52.728149 24705 hybrid_clock.cc:648] HybridClock initialized: now 1786056052728091 us; error 43 us; skew 500 ppm
I20260806 22:40:52.728562 24705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:52 dist-test-slave-zr1t krb5kdc[24687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056052, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:52.731953 24705 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:40:52.733820 24705 webserver.cc:460] Webserver started at http://127.23.100.190:45223/ using document root <none> and password file <none>
I20260806 22:40:52.734568 24705 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:52.734728 24705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:52.735085 24705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:52.738322 24705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "504e30cc0fc84a8cb911425345fe29e8"
format_stamp: "Formatted at 2026-08-06 22:40:52 on dist-test-slave-zr1t"
server_key: "1c1e94bfa9794ae928ef61d6c477847a"
server_key_iv: "3077a51dd46f5f5140df2aeb94f41e94"
server_key_version: "encryptionkey@0"
I20260806 22:40:52.739137 24705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "504e30cc0fc84a8cb911425345fe29e8"
format_stamp: "Formatted at 2026-08-06 22:40:52 on dist-test-slave-zr1t"
server_key: "1c1e94bfa9794ae928ef61d6c477847a"
server_key_iv: "3077a51dd46f5f5140df2aeb94f41e94"
server_key_version: "encryptionkey@0"
I20260806 22:40:52.744726 24705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:40:52.748257 24722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:52.749629 24705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:40:52.749828 24705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "504e30cc0fc84a8cb911425345fe29e8"
format_stamp: "Formatted at 2026-08-06 22:40:52 on dist-test-slave-zr1t"
server_key: "1c1e94bfa9794ae928ef61d6c477847a"
server_key_iv: "3077a51dd46f5f5140df2aeb94f41e94"
server_key_version: "encryptionkey@0"
I20260806 22:40:52.750051 24705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:52.778316 24705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:52.782823 24705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:52.783190 24705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:52.803025 24705 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:39395
I20260806 22:40:52.803062 24792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:39395 every 8 connection(s)
I20260806 22:40:52.804662 24705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:40:52.808538 24793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:52.814158 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24705
I20260806 22:40:52.814465 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:40:52.815133 23954 external_mini_cluster.cc:1428] Setting key 3634be95835360c302c54bfcee5dae50
I20260806 22:40:52.821111 24793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8: Bootstrap starting.
I20260806 22:40:52.825007 24793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:52.826184 24793 log.cc:824] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8: Log is configured to *not* fsync() on all Append() calls
I20260806 22:40:52.829312 24793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8: No bootstrap required, opened a new log
Aug 06 22:40:52 dist-test-slave-zr1t krb5kdc[24687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:52.840202 24793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e30cc0fc84a8cb911425345fe29e8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39395 } }
I20260806 22:40:52.840689 24793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:52.840814 24793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504e30cc0fc84a8cb911425345fe29e8, State: Initialized, Role: FOLLOWER
I20260806 22:40:52.841264 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:52.817441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.23.100.190:39395)
0806 22:40:52.818453 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:52.818470 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:52.818520 (+ 50us) server_negotiation.cc:407] Connection header received
0806 22:40:52.820920 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:52.820947 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:52.821472 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:52.821952 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:52.823354 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:52.825345 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:52.826731 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:52.827555 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:52.827747 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:52.832029 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:52.832042 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:52.832061 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:52.832112 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:52.836328 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:52.837069 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:52.837077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:52.837086 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:52.837368 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:52.837799 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:52.837804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:52.837808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:52.838608 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:52.838953 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:52.839407 (+ 454us) server_negotiation.cc:299] Negotiation successful
0806 22:40:52.839849 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":407,"threads_started":1}
I20260806 22:40:52.841487 24793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e30cc0fc84a8cb911425345fe29e8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39395 } }
I20260806 22:40:52.841833 24793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:40:52.842006 24793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:40:52.842195 24793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:52.844869 24793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e30cc0fc84a8cb911425345fe29e8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39395 } }
I20260806 22:40:52.845346 24793 leader_election.cc:302] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 504e30cc0fc84a8cb911425345fe29e8; no voters:
I20260806 22:40:52.845973 24793 leader_election.cc:288] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:40:52.846163 24800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:40:52.847095 24800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [term 1 LEADER]: Becoming Leader. State: Replica: 504e30cc0fc84a8cb911425345fe29e8, State: Running, Role: LEADER
I20260806 22:40:52.847657 24800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e30cc0fc84a8cb911425345fe29e8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39395 } }
I20260806 22:40:52.848212 24793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:40:52.851066 24803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 504e30cc0fc84a8cb911425345fe29e8. Latest consensus state: current_term: 1 leader_uuid: "504e30cc0fc84a8cb911425345fe29e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e30cc0fc84a8cb911425345fe29e8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39395 } } }
I20260806 22:40:52.851720 24803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [sys.catalog]: This master's current role is: LEADER
I20260806 22:40:52.851099 24801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "504e30cc0fc84a8cb911425345fe29e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504e30cc0fc84a8cb911425345fe29e8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39395 } } }
I20260806 22:40:52.852265 24801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8 [sys.catalog]: This master's current role is: LEADER
I20260806 22:40:52.853819 24811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:40:52.859419 24811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:40:52.870252 24811 catalog_manager.cc:1347] Generated new cluster ID: 96864ecd46e14c8d95c7d0e1f97bb822
I20260806 22:40:52.870702 24811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:40:52.888738 24811 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:40:52.890913 24811 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:40:52.904098 24811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 504e30cc0fc84a8cb911425345fe29e8: Generated new TSK 0
I20260806 22:40:52.905298 24811 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:40:52.961149 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39395
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:53.306540 24828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:53.306893 24828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:53.307085 24828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:53.314553 24828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:53.314713 24828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:53.314911 24828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:40:53.324803 24828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:53.326133 24828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:53.327461 24828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:53.340284 24838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:53.340289 24840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:53.340979 24836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:53.341455 24828 server_base.cc:1034] running on GCE node
I20260806 22:40:53.342077 24828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:53.343062 24828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:53.344260 24828 hybrid_clock.cc:648] HybridClock initialized: now 1786056053344214 us; error 37 us; skew 500 ppm
I20260806 22:40:53.344594 24828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:53 dist-test-slave-zr1t krb5kdc[24687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056053, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:53.347981 24828 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:40:53.349709 24828 webserver.cc:460] Webserver started at http://127.23.100.129:38673/ using document root <none> and password file <none>
I20260806 22:40:53.350445 24828 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:53.350596 24828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:53.350921 24828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:53.353910 24828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "cb6df98d13824ba1ba5f481e26ef9fa0"
format_stamp: "Formatted at 2026-08-06 22:40:53 on dist-test-slave-zr1t"
server_key: "5be25333f07e41ea61799874cfe7f057"
server_key_iv: "aa05957be4c8d6b0f25cc5c1ec236389"
server_key_version: "encryptionkey@0"
I20260806 22:40:53.354743 24828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "cb6df98d13824ba1ba5f481e26ef9fa0"
format_stamp: "Formatted at 2026-08-06 22:40:53 on dist-test-slave-zr1t"
server_key: "5be25333f07e41ea61799874cfe7f057"
server_key_iv: "aa05957be4c8d6b0f25cc5c1ec236389"
server_key_version: "encryptionkey@0"
I20260806 22:40:53.359967 24828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:40:53.363895 24853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:53.365334 24828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:40:53.365558 24828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "cb6df98d13824ba1ba5f481e26ef9fa0"
format_stamp: "Formatted at 2026-08-06 22:40:53 on dist-test-slave-zr1t"
server_key: "5be25333f07e41ea61799874cfe7f057"
server_key_iv: "aa05957be4c8d6b0f25cc5c1ec236389"
server_key_version: "encryptionkey@0"
I20260806 22:40:53.365777 24828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:53.382831 24828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:53.386900 24828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:53.387193 24828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:53.388159 24828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:40:53.389564 24828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:40:53.389670 24828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:53.389829 24828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:40:53.389889 24828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:53.424139 24828 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:37641
I20260806 22:40:53.424232 25001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:37641 every 8 connection(s)
I20260806 22:40:53.425652 24828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:40:53.434053 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24828
I20260806 22:40:53.434303 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:40:53.434934 23954 external_mini_cluster.cc:1428] Setting key 71c87919da546bc04b53b25ee5cdda7d
Aug 06 22:40:53 dist-test-slave-zr1t krb5kdc[24687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056053, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:53.446292 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:53.428141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:54991 (local address 127.23.100.190:39395)
0806 22:40:53.428636 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:53.428641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:53.429382 (+ 741us) server_negotiation.cc:407] Connection header received
0806 22:40:53.431303 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:53.431309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:53.431401 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:53.431521 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:53.433008 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:53.433947 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:53.435727 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:53.436136 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:53.436283 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:53.440263 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:53.440269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:53.440272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:53.440306 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:53.443072 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:53.443864 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:53.443885 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:53.443888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:53.444016 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:53.444471 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:53.444476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:53.444479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:53.444759 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:53.444930 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:53.445678 (+ 748us) server_negotiation.cc:299] Negotiation successful
0806 22:40:53.445923 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":259,"threads_started":1}
I20260806 22:40:53.447276 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:53.428507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:39395 (local address 127.23.100.129:54991)
0806 22:40:53.429217 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:53.429263 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:53.431026 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:53.431763 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:53.431770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:53.432273 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:53.432829 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:53.432846 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:53.434114 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:53.434118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:53.435560 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:53.435575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:53.436409 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:53.436419 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:53.436744 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:53.437503 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:53.437577 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:53.440013 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:53.443239 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:53.443248 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:53.443265 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:53.443735 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:53.444137 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:53.444142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:53.444146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:53.444348 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:53.444895 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:53.444904 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:53.445260 (+ 356us) client_negotiation.cc:769] Sending connection context
0806 22:40:53.445570 (+ 310us) client_negotiation.cc:240] Negotiation successful
0806 22:40:53.446111 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":280,"threads_started":1}
I20260806 22:40:53.449015 25002 heartbeater.cc:347] Connected to a master server at 127.23.100.190:39395
I20260806 22:40:53.449348 25002 heartbeater.cc:464] Registering TS with master...
I20260806 22:40:53.450227 25002 heartbeater.cc:511] Master 127.23.100.190:39395 requested a full tablet report, sending...
I20260806 22:40:53.452672 24747 ts_manager.cc:194] Registered new tserver with Master: cb6df98d13824ba1ba5f481e26ef9fa0 (127.23.100.129:37641)
I20260806 22:40:53.454906 24747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:54991
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:40:53.484359 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39395
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:53.817332 25009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:53.817715 25009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:53.817898 25009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:53.825229 25009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:53.825397 25009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:53.825656 25009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:40:53.836539 25009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:53.838092 25009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:53.839557 25009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:53.853484 25014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:53.853396 25017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:53.853441 25019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:53.854612 25009 server_base.cc:1034] running on GCE node
I20260806 22:40:53.855252 25009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:53.856225 25009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:53.857466 25009 hybrid_clock.cc:648] HybridClock initialized: now 1786056053857412 us; error 52 us; skew 500 ppm
I20260806 22:40:53.857820 25009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:53 dist-test-slave-zr1t krb5kdc[24687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056053, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:53.865913 25009 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:40:53.867453 25009 webserver.cc:460] Webserver started at http://127.23.100.130:46151/ using document root <none> and password file <none>
I20260806 22:40:53.868119 25009 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:53.868220 25009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:53.868497 25009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:53.871752 25009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "2a7bdac6cf244e618131c32deb4f2817"
format_stamp: "Formatted at 2026-08-06 22:40:53 on dist-test-slave-zr1t"
server_key: "bd5184a0f574f256c167aed275569106"
server_key_iv: "efaff47b7a1a272c9404dc32962f692a"
server_key_version: "encryptionkey@0"
I20260806 22:40:53.872532 25009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "2a7bdac6cf244e618131c32deb4f2817"
format_stamp: "Formatted at 2026-08-06 22:40:53 on dist-test-slave-zr1t"
server_key: "bd5184a0f574f256c167aed275569106"
server_key_iv: "efaff47b7a1a272c9404dc32962f692a"
server_key_version: "encryptionkey@0"
I20260806 22:40:53.878062 25009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 22:40:53.881862 25030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:53.883342 25009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 22:40:53.883512 25009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "2a7bdac6cf244e618131c32deb4f2817"
format_stamp: "Formatted at 2026-08-06 22:40:53 on dist-test-slave-zr1t"
server_key: "bd5184a0f574f256c167aed275569106"
server_key_iv: "efaff47b7a1a272c9404dc32962f692a"
server_key_version: "encryptionkey@0"
I20260806 22:40:53.883699 25009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:53.914966 25009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:53.919762 25009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:53.920089 25009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:53.921131 25009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:40:53.922674 25009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:40:53.922787 25009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:53.922945 25009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:40:53.923022 25009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:53.959152 25009 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:39329
I20260806 22:40:53.959216 25181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:39329 every 8 connection(s)
I20260806 22:40:53.960573 25009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:40:53.966434 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 25009
I20260806 22:40:53.966660 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:40:53.967257 23954 external_mini_cluster.cc:1428] Setting key 977bae8adf5ed87ceb4d84f85f7cbb2c
Aug 06 22:40:53 dist-test-slave-zr1t krb5kdc[24687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056053, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:53.982003 25189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:53.963208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:60755 (local address 127.23.100.190:39395)
0806 22:40:53.963709 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:53.963713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:53.964502 (+ 789us) server_negotiation.cc:407] Connection header received
0806 22:40:53.966799 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:53.966805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:53.966890 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:53.967001 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:53.968949 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:53.970128 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:53.971703 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:53.972476 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:53.972624 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:53.976081 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:53.976090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:53.976098 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:53.976146 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:53.978882 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:53.979619 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:53.979627 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:53.979632 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:53.979766 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:53.980196 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:53.980203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:53.980206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:53.980479 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:53.980636 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:53.981344 (+ 708us) server_negotiation.cc:299] Negotiation successful
0806 22:40:53.981621 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":278,"threads_started":1}
I20260806 22:40:53.984227 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:53.963635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:39395 (local address 127.23.100.130:60755)
0806 22:40:53.964335 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:53.964374 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:53.966442 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:53.967293 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:53.967306 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:53.968024 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:53.968719 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:53.968766 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:53.970289 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:53.970293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:53.971548 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:53.971562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:53.972703 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:53.972708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:53.972908 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:53.973655 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:53.973677 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:53.975840 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:53.979041 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:53.979050 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:53.979067 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:53.979496 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:53.979880 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:53.979885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:53.979889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:53.980085 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:53.980609 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:53.980617 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:53.980986 (+ 369us) client_negotiation.cc:769] Sending connection context
0806 22:40:53.981262 (+ 276us) client_negotiation.cc:240] Negotiation successful
0806 22:40:53.981813 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":285,"threads_started":1}
I20260806 22:40:53.985853 25184 heartbeater.cc:347] Connected to a master server at 127.23.100.190:39395
I20260806 22:40:53.986223 25184 heartbeater.cc:464] Registering TS with master...
I20260806 22:40:53.987293 25184 heartbeater.cc:511] Master 127.23.100.190:39395 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
I20260806 22:40:53.989012 24747 ts_manager.cc:194] Registered new tserver with Master: 2a7bdac6cf244e618131c32deb4f2817 (127.23.100.130:39329)
I20260806 22:40:53.989959 24747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:60755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:40:54.016294 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39395
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:54.359094 25197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:54.359408 25197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:54.359575 25197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:54.367260 25197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:54.367390 25197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:54.367576 25197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:40:54.378055 25197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:54.379462 25197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:54.380865 25197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:54.395493 25207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:54.395404 25209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:54.395473 25211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:54.396420 25197 server_base.cc:1034] running on GCE node
I20260806 22:40:54.397076 25197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:54.398200 25197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:54.399458 25197 hybrid_clock.cc:648] HybridClock initialized: now 1786056054399380 us; error 54 us; skew 500 ppm
I20260806 22:40:54.399887 25197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:54 dist-test-slave-zr1t krb5kdc[24687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056054, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:54.403738 25197 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:40:54.405716 25197 webserver.cc:460] Webserver started at http://127.23.100.131:33699/ using document root <none> and password file <none>
I20260806 22:40:54.406786 25197 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:54.406980 25197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:54.407357 25197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:54.410817 25197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "b867ca61876f45fe8224298d9b90cb4b"
format_stamp: "Formatted at 2026-08-06 22:40:54 on dist-test-slave-zr1t"
server_key: "39cc4e32e714d531ffc423ce549fb2fd"
server_key_iv: "591750ca73a6c3b8c9d57c9c4fa5efc5"
server_key_version: "encryptionkey@0"
I20260806 22:40:54.411688 25197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "b867ca61876f45fe8224298d9b90cb4b"
format_stamp: "Formatted at 2026-08-06 22:40:54 on dist-test-slave-zr1t"
server_key: "39cc4e32e714d531ffc423ce549fb2fd"
server_key_iv: "591750ca73a6c3b8c9d57c9c4fa5efc5"
server_key_version: "encryptionkey@0"
I20260806 22:40:54.417397 25197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260806 22:40:54.421296 25220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:54.422912 25197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:40:54.423139 25197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "b867ca61876f45fe8224298d9b90cb4b"
format_stamp: "Formatted at 2026-08-06 22:40:54 on dist-test-slave-zr1t"
server_key: "39cc4e32e714d531ffc423ce549fb2fd"
server_key_iv: "591750ca73a6c3b8c9d57c9c4fa5efc5"
server_key_version: "encryptionkey@0"
I20260806 22:40:54.423442 25197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:54.450546 25197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:54.455710 25197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:54.456018 25197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:54.457145 25197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:40:54.458287 25002 heartbeater.cc:503] Master 127.23.100.190:39395 was elected leader, sending a full tablet report...
I20260806 22:40:54.459009 25197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:40:54.459146 25197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:54.459335 25197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:40:54.459405 25197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:54.498600 25197 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:40121
I20260806 22:40:54.498658 25376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:40121 every 8 connection(s)
I20260806 22:40:54.500128 25197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:40:54.508651 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 25197
I20260806 22:40:54.508873 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:40:54.509299 23954 external_mini_cluster.cc:1428] Setting key 13e66418cd3eff1bd5ee09e47eb598d7
Aug 06 22:40:54 dist-test-slave-zr1t krb5kdc[24687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056054, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:54.522827 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.502856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:55167 (local address 127.23.100.190:39395)
0806 22:40:54.503358 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:54.503364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:54.504411 (+ 1047us) server_negotiation.cc:407] Connection header received
0806 22:40:54.506685 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:54.506692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:54.506824 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:54.506964 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:54.508841 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.509933 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.512070 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.512590 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.512731 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.516618 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:54.516623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:54.516627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:54.516664 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:54.519558 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:54.520473 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:54.520477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:54.520479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:54.520600 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:54.521139 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:54.521142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:54.521144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:54.521407 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:54.521644 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:54.522343 (+ 699us) server_negotiation.cc:299] Negotiation successful
0806 22:40:54.522545 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":262,"threads_started":1}
I20260806 22:40:54.524062 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.503427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:39395 (local address 127.23.100.131:55167)
0806 22:40:54.504207 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:54.504263 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:54.506347 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:54.507240 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:54.507262 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:54.507948 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:54.508621 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.508642 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.510124 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.510129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.511914 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.511932 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.512823 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.512831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.513096 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.513929 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:54.513960 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:54.516369 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:54.519774 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:54.519784 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:54.519799 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:54.520342 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:54.520804 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:54.520808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:54.520810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:54.521033 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:54.521619 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:54.521626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:54.521986 (+ 360us) client_negotiation.cc:769] Sending connection context
0806 22:40:54.522240 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 22:40:54.522704 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":287,"threads_started":1}
I20260806 22:40:54.525607 25378 heartbeater.cc:347] Connected to a master server at 127.23.100.190:39395
I20260806 22:40:54.525951 25378 heartbeater.cc:464] Registering TS with master...
I20260806 22:40:54.526680 25378 heartbeater.cc:511] Master 127.23.100.190:39395 requested a full tablet report, sending...
I20260806 22:40:54.528189 24745 ts_manager.cc:194] Registered new tserver with Master: b867ca61876f45fe8224298d9b90cb4b (127.23.100.131:40121)
I20260806 22:40:54.529196 24745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:55167
I20260806 22:40:54.538335 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:40:54.560168 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.545569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.23.100.190:39395)
0806 22:40:54.545759 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:54.545764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:54.546390 (+ 626us) server_negotiation.cc:407] Connection header received
0806 22:40:54.546532 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:54.546537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:54.546609 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:54.546706 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:54.548418 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.549221 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.550727 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.551162 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.551269 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.554118 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:54.554124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:54.554127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:54.554166 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:54.557209 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:54.558087 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:54.558091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:54.558094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:54.558227 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:54.558651 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:54.558654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:54.558656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:54.558993 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:54.559184 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:54.559573 (+ 389us) server_negotiation.cc:299] Negotiation successful
0806 22:40:54.559751 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 22:40:54.567239 24745 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:40:54.574796 24745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:40:54.600917 25395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.588882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:37641 (local address 127.0.0.1:37810)
0806 22:40:54.589776 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:54.589822 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:54.590050 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:54.593619 (+ 3569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:54.593624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:54.593656 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:54.594125 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.594140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.596152 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.596157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.598242 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.598263 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.599905 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.599913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.600150 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.600209 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 22:40:54.600329 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 22:40:54.600525 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":3304,"threads_started":1}
I20260806 22:40:54.601735 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.589713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.23.100.129:37641)
0806 22:40:54.593084 (+ 3371us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:54.593092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:54.593116 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:40:54.593211 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:54.593216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:54.593386 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:54.593625 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:54.594333 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.595980 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.598419 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.599659 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.600870 (+ 1211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.601067 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:54.601153 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 22:40:54.601375 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3185,"thread_start_us":3192,"threads_started":1}
I20260806 22:40:54.602393 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.590123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.23.100.131:40121)
0806 22:40:54.590796 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:54.590806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:54.591474 (+ 668us) server_negotiation.cc:407] Connection header received
0806 22:40:54.591820 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:54.591833 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:54.592149 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:54.592343 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:54.594240 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.595529 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.599087 (+ 3558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.600594 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.600784 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.600977 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:54.601846 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 22:40:54.602034 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":347,"threads_started":1}
I20260806 22:40:54.603219 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.590639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:39329 (local address 127.0.0.1:46560)
0806 22:40:54.592308 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:54.592330 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:54.592457 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:54.593912 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:54.593915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:54.593939 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:54.594529 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.594544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.596898 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.596906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.599970 (+ 3064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.599984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.602313 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.602320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.602515 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.602554 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 22:40:54.602674 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 22:40:54.602811 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1518,"mutex_wait_us":48,"thread_start_us":376,"threads_started":1}
I20260806 22:40:54.603253 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.591074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.23.100.130:39329)
0806 22:40:54.592990 (+ 1916us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:54.592999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:54.593030 (+ 31us) server_negotiation.cc:407] Connection header received
0806 22:40:54.593150 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:54.593158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:54.593386 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:54.593625 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:54.594717 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.596768 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.600172 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.602071 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.602284 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.602532 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:54.602740 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 22:40:54.602918 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1750,"thread_start_us":1632,"threads_started":1}
I20260806 22:40:54.605716 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.589088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:40121 (local address 127.0.0.1:32880)
0806 22:40:54.591337 (+ 2249us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:54.591366 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:54.591607 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:54.593364 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:54.593385 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:54.593419 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:54.594000 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.594020 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.595716 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.595721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.598897 (+ 3176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.598914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.601465 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.601472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.601650 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.601674 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 22:40:54.601776 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 22:40:54.603229 (+ 1453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2091,"mutex_wait_us":77,"thread_start_us":2908,"threads_started":1}
I20260806 22:40:54.606247 24915 tablet_service.cc:1467] Processing CreateTablet for tablet 72a4e929f9ce47d1bbe5afaeea7f7204 (DEFAULT_TABLE table=test-table [id=59237330f98148b6bb41ab023111bcc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:40:54.607164 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 72a4e929f9ce47d1bbe5afaeea7f7204 (DEFAULT_TABLE table=test-table [id=59237330f98148b6bb41ab023111bcc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:40:54.607939 24915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a4e929f9ce47d1bbe5afaeea7f7204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:54.608554 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a4e929f9ce47d1bbe5afaeea7f7204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:54.612479 25284 tablet_service.cc:1467] Processing CreateTablet for tablet 72a4e929f9ce47d1bbe5afaeea7f7204 (DEFAULT_TABLE table=test-table [id=59237330f98148b6bb41ab023111bcc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:40:54.613721 25284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a4e929f9ce47d1bbe5afaeea7f7204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:54.623984 25403 tablet_bootstrap.cc:492] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0: Bootstrap starting.
I20260806 22:40:54.624986 25402 tablet_bootstrap.cc:492] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817: Bootstrap starting.
I20260806 22:40:54.626091 25404 tablet_bootstrap.cc:492] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b: Bootstrap starting.
I20260806 22:40:54.629441 25402 tablet_bootstrap.cc:654] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:54.629451 25404 tablet_bootstrap.cc:654] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:54.630766 25402 log.cc:824] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817: Log is configured to *not* fsync() on all Append() calls
I20260806 22:40:54.630824 25404 log.cc:824] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b: Log is configured to *not* fsync() on all Append() calls
I20260806 22:40:54.631503 25403 tablet_bootstrap.cc:654] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:54.632768 25403 log.cc:824] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0: Log is configured to *not* fsync() on all Append() calls
I20260806 22:40:54.634611 25402 tablet_bootstrap.cc:492] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817: No bootstrap required, opened a new log
I20260806 22:40:54.634905 25402 ts_tablet_manager.cc:1397] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 22:40:54.635890 25403 tablet_bootstrap.cc:492] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0: No bootstrap required, opened a new log
I20260806 22:40:54.636160 25404 tablet_bootstrap.cc:492] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b: No bootstrap required, opened a new log
I20260806 22:40:54.636160 25403 ts_tablet_manager.cc:1397] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260806 22:40:54.636436 25404 ts_tablet_manager.cc:1397] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260806 22:40:54.643432 25402 raft_consensus.cc:348] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37641 } } peers { permanent_uuid: "b867ca61876f45fe8224298d9b90cb4b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 40121 } } peers { permanent_uuid: "2a7bdac6cf244e618131c32deb4f2817" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39329 } }
I20260806 22:40:54.643929 25402 raft_consensus.cc:374] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:54.644035 25402 raft_consensus.cc:729] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7bdac6cf244e618131c32deb4f2817, State: Initialized, Role: FOLLOWER
I20260806 22:40:54.644822 25402 consensus_queue.cc:260] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37641 } } peers { permanent_uuid: "b867ca61876f45fe8224298d9b90cb4b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 40121 } } peers { permanent_uuid: "2a7bdac6cf244e618131c32deb4f2817" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39329 } }
I20260806 22:40:54.647441 25403 raft_consensus.cc:348] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37641 } } peers { permanent_uuid: "b867ca61876f45fe8224298d9b90cb4b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 40121 } } peers { permanent_uuid: "2a7bdac6cf244e618131c32deb4f2817" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39329 } }
I20260806 22:40:54.647491 25404 raft_consensus.cc:348] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37641 } } peers { permanent_uuid: "b867ca61876f45fe8224298d9b90cb4b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 40121 } } peers { permanent_uuid: "2a7bdac6cf244e618131c32deb4f2817" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39329 } }
I20260806 22:40:54.647921 25404 raft_consensus.cc:374] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:54.647922 25403 raft_consensus.cc:374] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:54.648029 25403 raft_consensus.cc:729] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6df98d13824ba1ba5f481e26ef9fa0, State: Initialized, Role: FOLLOWER
I20260806 22:40:54.648029 25404 raft_consensus.cc:729] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b867ca61876f45fe8224298d9b90cb4b, State: Initialized, Role: FOLLOWER
I20260806 22:40:54.648804 25403 consensus_queue.cc:260] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37641 } } peers { permanent_uuid: "b867ca61876f45fe8224298d9b90cb4b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 40121 } } peers { permanent_uuid: "2a7bdac6cf244e618131c32deb4f2817" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39329 } }
I20260806 22:40:54.649160 25404 consensus_queue.cc:260] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37641 } } peers { permanent_uuid: "b867ca61876f45fe8224298d9b90cb4b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 40121 } } peers { permanent_uuid: "2a7bdac6cf244e618131c32deb4f2817" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39329 } }
I20260806 22:40:54.651252 25403 ts_tablet_manager.cc:1428] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260806 22:40:54.652426 25184 heartbeater.cc:503] Master 127.23.100.190:39395 was elected leader, sending a full tablet report...
I20260806 22:40:54.653035 25402 ts_tablet_manager.cc:1428] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260806 22:40:54.654846 25378 heartbeater.cc:503] Master 127.23.100.190:39395 was elected leader, sending a full tablet report...
I20260806 22:40:54.655578 25404 ts_tablet_manager.cc:1428] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b: Time spent starting tablet: real 0.019s user 0.017s sys 0.002s
W20260806 22:40:54.678190 25003 tablet.cc:2369] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:40:54.712872 25188 tablet.cc:2369] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:40:54.752028 25379 tablet.cc:2369] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:40:54.871824 25409 raft_consensus.cc:482] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:40:54.872121 25409 raft_consensus.cc:504] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37641 } } peers { permanent_uuid: "b867ca61876f45fe8224298d9b90cb4b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 40121 } } peers { permanent_uuid: "2a7bdac6cf244e618131c32deb4f2817" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39329 } }
I20260806 22:40:54.873939 25409 leader_election.cc:288] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b867ca61876f45fe8224298d9b90cb4b (127.23.100.131:40121), 2a7bdac6cf244e618131c32deb4f2817 (127.23.100.130:39329)
I20260806 22:40:54.883443 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.874562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:36961 (local address 127.23.100.131:40121)
0806 22:40:54.874758 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:54.874767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:54.875630 (+ 863us) server_negotiation.cc:407] Connection header received
0806 22:40:54.875812 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:54.875816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:54.875879 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:54.875964 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:54.876655 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.877881 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.880860 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.882180 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.882378 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.882449 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:54.883028 (+ 579us) server_negotiation.cc:299] Negotiation successful
0806 22:40:54.883171 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 22:40:54.883443 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.874430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:41663 (local address 127.23.100.130:39329)
0806 22:40:54.874674 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:54.874681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:54.875578 (+ 897us) server_negotiation.cc:407] Connection header received
0806 22:40:54.876132 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:54.876137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:54.876194 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:54.876264 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:54.876964 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.878047 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.880870 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.882178 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.882371 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.882449 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:54.883028 (+ 579us) server_negotiation.cc:299] Negotiation successful
0806 22:40:54.883171 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 22:40:54.883651 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.874473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:40121 (local address 127.23.100.129:36961)
0806 22:40:54.875473 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:54.875488 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:54.875656 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:54.875989 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:54.875992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:54.876022 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:54.876431 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.876449 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.878027 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.878031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.880649 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.880666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.882467 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.882480 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.882698 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.882825 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 22:40:54.882940 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 22:40:54.883116 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":111,"spinlock_wait_cycles":25856,"thread_start_us":418,"threads_started":1}
I20260806 22:40:54.884078 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.874435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:39329 (local address 127.23.100.129:41663)
0806 22:40:54.875437 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:54.875463 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:54.875656 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:54.876360 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:54.876370 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:54.876397 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:54.876778 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.876788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.878204 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.878213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.880649 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:54.880666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.882499 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.882504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:54.882822 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.882848 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 22:40:54.882940 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 22:40:54.883116 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":189,"spinlock_wait_cycles":14080,"thread_start_us":661,"threads_started":1}
I20260806 22:40:54.884881 25311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a4e929f9ce47d1bbe5afaeea7f7204" candidate_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b867ca61876f45fe8224298d9b90cb4b" is_pre_election: true
I20260806 22:40:54.885128 25125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a4e929f9ce47d1bbe5afaeea7f7204" candidate_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7bdac6cf244e618131c32deb4f2817" is_pre_election: true
I20260806 22:40:54.885429 25311 raft_consensus.cc:2454] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6df98d13824ba1ba5f481e26ef9fa0 in term 0.
I20260806 22:40:54.885710 25125 raft_consensus.cc:2454] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6df98d13824ba1ba5f481e26ef9fa0 in term 0.
I20260806 22:40:54.886260 24857 leader_election.cc:302] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b867ca61876f45fe8224298d9b90cb4b, cb6df98d13824ba1ba5f481e26ef9fa0; no voters:
I20260806 22:40:54.886715 25409 raft_consensus.cc:2781] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:40:54.886886 25409 raft_consensus.cc:482] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:40:54.887013 25409 raft_consensus.cc:3037] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:54.889729 25409 raft_consensus.cc:504] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37641 } } peers { permanent_uuid: "b867ca61876f45fe8224298d9b90cb4b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 40121 } } peers { permanent_uuid: "2a7bdac6cf244e618131c32deb4f2817" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39329 } }
I20260806 22:40:54.890578 25409 leader_election.cc:288] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [CANDIDATE]: Term 1 election: Requested vote from peers b867ca61876f45fe8224298d9b90cb4b (127.23.100.131:40121), 2a7bdac6cf244e618131c32deb4f2817 (127.23.100.130:39329)
I20260806 22:40:54.891083 25125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a4e929f9ce47d1bbe5afaeea7f7204" candidate_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7bdac6cf244e618131c32deb4f2817"
I20260806 22:40:54.891086 25311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a4e929f9ce47d1bbe5afaeea7f7204" candidate_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b867ca61876f45fe8224298d9b90cb4b"
I20260806 22:40:54.891288 25125 raft_consensus.cc:3037] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:54.891305 25311 raft_consensus.cc:3037] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:54.893432 25125 raft_consensus.cc:2454] T 72a4e929f9ce47d1bbe5afaeea7f7204 P 2a7bdac6cf244e618131c32deb4f2817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6df98d13824ba1ba5f481e26ef9fa0 in term 1.
I20260806 22:40:54.893467 25311 raft_consensus.cc:2454] T 72a4e929f9ce47d1bbe5afaeea7f7204 P b867ca61876f45fe8224298d9b90cb4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6df98d13824ba1ba5f481e26ef9fa0 in term 1.
I20260806 22:40:54.894050 24856 leader_election.cc:302] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7bdac6cf244e618131c32deb4f2817, cb6df98d13824ba1ba5f481e26ef9fa0; no voters:
I20260806 22:40:54.894397 25409 raft_consensus.cc:2781] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:40:54.894804 25409 raft_consensus.cc:686] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [term 1 LEADER]: Becoming Leader. State: Replica: cb6df98d13824ba1ba5f481e26ef9fa0, State: Running, Role: LEADER
I20260806 22:40:54.895298 25409 consensus_queue.cc:237] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37641 } } peers { permanent_uuid: "b867ca61876f45fe8224298d9b90cb4b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 40121 } } peers { permanent_uuid: "2a7bdac6cf244e618131c32deb4f2817" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39329 } }
I20260806 22:40:54.899699 24747 catalog_manager.cc:5495] T 72a4e929f9ce47d1bbe5afaeea7f7204 P cb6df98d13824ba1ba5f481e26ef9fa0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb6df98d13824ba1ba5f481e26ef9fa0 (127.23.100.129). New cstate: current_term: 1 leader_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6df98d13824ba1ba5f481e26ef9fa0" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b867ca61876f45fe8224298d9b90cb4b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 40121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a7bdac6cf244e618131c32deb4f2817" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39329 } health_report { overall_health: UNKNOWN } } }
Aug 06 22:40:54 dist-test-slave-zr1t krb5kdc[24687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:40:54.979678 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:54.962692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.23.100.129:37641)
0806 22:40:54.962935 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:54.962942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:54.963520 (+ 578us) server_negotiation.cc:407] Connection header received
0806 22:40:54.963691 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:54.963695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:54.963765 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:54.963893 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:54.965911 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.966641 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.968555 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:54.969037 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:54.969181 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:54.974228 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:54.974245 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:54.974260 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:54.974301 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:54.977079 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:54.977862 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:54.977886 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:54.977888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:54.978058 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:54.978381 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:54.978384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:54.978386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:54.978738 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:54.978981 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:54.979228 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 22:40:54.979435 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 22:40:55 dist-test-slave-zr1t krb5kdc[24687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056055, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 22:40:55 dist-test-slave-zr1t krb5kdc[24687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056055, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:40:55.036734 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:55.022260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.23.100.129:37641)
0806 22:40:55.022568 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:55.022576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:55.022894 (+ 318us) server_negotiation.cc:407] Connection header received
0806 22:40:55.023129 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:55.023133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:55.023206 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:55.023324 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:55.024813 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:55.025622 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:55.026922 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:55.027353 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:55.027493 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:55.031718 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:55.031723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:55.031727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:55.031763 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:55.034422 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:55.035056 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:55.035060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:55.035062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:55.035195 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:55.035553 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:55.035556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:55.035558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:55.035815 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:55.036034 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:55.036262 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 22:40:55.036480 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
Aug 06 22:40:55 dist-test-slave-zr1t krb5kdc[24687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 22:40:55 dist-test-slave-zr1t krb5kdc[24687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:40:55.094447 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:55.079647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.23.100.129:37641)
0806 22:40:55.079919 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:55.079927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:55.080201 (+ 274us) server_negotiation.cc:407] Connection header received
0806 22:40:55.080438 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:55.080442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:55.080526 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:55.080642 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:55.082068 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:55.082862 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:55.084157 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:55.084608 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:55.084753 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:55.088950 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:55.088955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:55.088959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:55.089001 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:55.091597 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:55.092350 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:55.092357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:55.092361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:55.092506 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:55.092917 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:55.092924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:55.092928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:55.093212 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:55.093498 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:55.093820 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 22:40:55.094081 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260806 22:40:55.096292 24887 tablet_service.cc:2908] Rejecting scan request for tablet 72a4e929f9ce47d1bbe5afaeea7f7204: Uninitialized: clean time has not yet been initialized
I20260806 22:40:55.099874 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24828
I20260806 22:40:55.113282 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:55.100841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.23.100.129:37641)
0806 22:40:55.101048 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:55.101053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:55.101473 (+ 420us) server_negotiation.cc:407] Connection header received
0806 22:40:55.101666 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:55.101672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:55.101775 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:55.101905 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:55.103166 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:55.103852 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:55.105067 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:55.105486 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:55.105666 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:55.108042 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:55.108046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:55.108049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:55.108082 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:55.110814 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:55.111543 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:55.111547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:55.111549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:55.111683 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:55.112040 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:55.112044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:55.112046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:55.112328 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:55.112566 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:55.112808 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 22:40:55.113029 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 22:40:55.114054 24992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:40:55.207549 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24828
I20260806 22:40:55.232087 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25009
Aug 06 22:40:55 dist-test-slave-zr1t krb5kdc[24687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:40:55.247161 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:55.232875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.23.100.130:39329)
0806 22:40:55.233056 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:55.233061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:55.233078 (+ 17us) server_negotiation.cc:407] Connection header received
0806 22:40:55.233249 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:55.233253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:55.233319 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:55.233405 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:55.234715 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:55.235438 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:55.236811 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:55.237192 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:55.237287 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:55.241628 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:55.241646 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:55.241661 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:55.241706 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:55.244650 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:55.245330 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:55.245338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:55.245340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:55.245491 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:55.245878 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:55.245881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:55.245883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:55.246249 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:55.246505 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:55.246723 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 22:40:55.246897 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 22:40:55.247860 25177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:40:55.336870 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 25009
I20260806 22:40:55.362383 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25197
Aug 06 22:40:55 dist-test-slave-zr1t krb5kdc[24687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:40:55.377682 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:55.363224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.23.100.131:40121)
0806 22:40:55.363412 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:55.363417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:55.363437 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:40:55.363593 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:55.363597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:55.363672 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:55.363759 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:55.365000 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:55.365802 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:55.367116 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:55.367497 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:55.367586 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:55.371784 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:55.371801 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:55.371818 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:55.371857 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:55.374784 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:55.375496 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:55.375512 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:55.375515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:55.375657 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:55.376036 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:55.376042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:55.376045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:55.376396 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:55.376702 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:55.377013 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 22:40:55.377248 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:40:55.378355 25371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:40:55.472111 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 25197
I20260806 22:40:55.497658 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24705
I20260806 22:40:55.498992 24787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:40:55.614777 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24705
2026-08-06T22:40:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4761 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:40:55 dist-test-slave-zr1t krb5kdc[25443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:40:55 dist-test-slave-zr1t krb5kdc[25443](info): set up 2 sockets
Aug 06 22:40:55 dist-test-slave-zr1t krb5kdc[25443](info): commencing operation
krb5kdc: starting...
W20260806 22:40:56.902758 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:40:57 dist-test-slave-zr1t krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:40:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:40:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:40:57.110741 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:42951
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:42951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:57.440521 25461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:57.440802 25461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:57.440946 25461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:57.448215 25461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:40:57.448315 25461 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 22:40:57.448386 25461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:57.448498 25461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:57.448611 25461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:40:57.448694 25461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:40:57.459048 25461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:42951
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:42951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:57.460244 25461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:57.461496 25461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:57.473207 25468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:57.473317 25469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:57.473816 25471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:57.475064 25461 server_base.cc:1034] running on GCE node
I20260806 22:40:57.475785 25461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:57.477139 25461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:57.478358 25461 hybrid_clock.cc:648] HybridClock initialized: now 1786056057478297 us; error 37 us; skew 500 ppm
I20260806 22:40:57.478649 25461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:57 dist-test-slave-zr1t krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056057, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:57.481909 25461 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:40:57.483404 25461 webserver.cc:460] Webserver started at http://127.23.100.190:44239/ using document root <none> and password file <none>
I20260806 22:40:57.484115 25461 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:57.484248 25461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:57.484587 25461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:57.487603 25461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "b75f26fbde6c4aa19baabcc700172a3a"
format_stamp: "Formatted at 2026-08-06 22:40:57 on dist-test-slave-zr1t"
server_key: "521b209044698050e67748c2d06b3e10"
server_key_iv: "b27dd993fcddbc7e1691340a4efe53c9"
server_key_version: "encryptionkey@0"
I20260806 22:40:57.488379 25461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "b75f26fbde6c4aa19baabcc700172a3a"
format_stamp: "Formatted at 2026-08-06 22:40:57 on dist-test-slave-zr1t"
server_key: "521b209044698050e67748c2d06b3e10"
server_key_iv: "b27dd993fcddbc7e1691340a4efe53c9"
server_key_version: "encryptionkey@0"
I20260806 22:40:57.493471 25461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 22:40:57.496944 25479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:57.498240 25461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:40:57.498413 25461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "b75f26fbde6c4aa19baabcc700172a3a"
format_stamp: "Formatted at 2026-08-06 22:40:57 on dist-test-slave-zr1t"
server_key: "521b209044698050e67748c2d06b3e10"
server_key_iv: "b27dd993fcddbc7e1691340a4efe53c9"
server_key_version: "encryptionkey@0"
I20260806 22:40:57.498631 25461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:57.523185 25461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:57.527714 25461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:57.528007 25461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:57.550540 25461 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:42951
I20260806 22:40:57.550565 25569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:42951 every 8 connection(s)
I20260806 22:40:57.551975 25461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:40:57.555683 25571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:57.561902 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 25461
I20260806 22:40:57.562166 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:40:57.563109 23954 external_mini_cluster.cc:1428] Setting key 78310aba6e43aa7acc5d62e8fa41143a
I20260806 22:40:57.567173 25571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a: Bootstrap starting.
I20260806 22:40:57.570931 25571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:57.572014 25571 log.cc:824] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a: Log is configured to *not* fsync() on all Append() calls
I20260806 22:40:57.575107 25571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a: No bootstrap required, opened a new log
Aug 06 22:40:57 dist-test-slave-zr1t krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:57.583742 25571 raft_consensus.cc:348] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f26fbde6c4aa19baabcc700172a3a" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42951 } }
I20260806 22:40:57.584167 25571 raft_consensus.cc:374] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:57.584285 25571 raft_consensus.cc:729] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75f26fbde6c4aa19baabcc700172a3a, State: Initialized, Role: FOLLOWER
I20260806 22:40:57.584985 25571 consensus_queue.cc:260] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f26fbde6c4aa19baabcc700172a3a" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42951 } }
I20260806 22:40:57.585243 25571 raft_consensus.cc:388] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:40:57.585408 25571 raft_consensus.cc:482] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:40:57.585644 25571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:57.586975 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:57.565338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.23.100.190:42951)
0806 22:40:57.566239 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:57.566251 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:57.566294 (+ 43us) server_negotiation.cc:407] Connection header received
0806 22:40:57.568070 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:57.568093 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:57.568479 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:57.568795 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:57.570659 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:57.572020 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:57.573660 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:57.574312 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:57.574438 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:57.579329 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:57.579337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:57.579372 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:57.579412 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:57.582704 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:57.583471 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:57.583478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:57.583485 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:57.583659 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:57.584065 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:57.584069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:57.584071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:57.584752 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:57.585083 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:57.585511 (+ 428us) server_negotiation.cc:299] Negotiation successful
0806 22:40:57.585984 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":445,"threads_started":1}
I20260806 22:40:57.588181 25571 raft_consensus.cc:504] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f26fbde6c4aa19baabcc700172a3a" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42951 } }
I20260806 22:40:57.588708 25571 leader_election.cc:302] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75f26fbde6c4aa19baabcc700172a3a; no voters:
I20260806 22:40:57.589318 25571 leader_election.cc:288] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:40:57.589511 25576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:40:57.590817 25576 raft_consensus.cc:686] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [term 1 LEADER]: Becoming Leader. State: Replica: b75f26fbde6c4aa19baabcc700172a3a, State: Running, Role: LEADER
I20260806 22:40:57.591313 25576 consensus_queue.cc:237] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f26fbde6c4aa19baabcc700172a3a" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42951 } }
I20260806 22:40:57.591535 25571 sys_catalog.cc:564] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:40:57.593864 25579 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b75f26fbde6c4aa19baabcc700172a3a. Latest consensus state: current_term: 1 leader_uuid: "b75f26fbde6c4aa19baabcc700172a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f26fbde6c4aa19baabcc700172a3a" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42951 } } }
I20260806 22:40:57.594282 25579 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [sys.catalog]: This master's current role is: LEADER
I20260806 22:40:57.594460 25578 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b75f26fbde6c4aa19baabcc700172a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f26fbde6c4aa19baabcc700172a3a" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42951 } } }
I20260806 22:40:57.594784 25578 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a [sys.catalog]: This master's current role is: LEADER
I20260806 22:40:57.597673 25584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:40:57.602687 25584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:40:57.612022 25584 catalog_manager.cc:1347] Generated new cluster ID: 6201e1412e8548cca14fa984ba48fcb2
I20260806 22:40:57.612151 25584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:40:57.622722 25584 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:40:57.624096 25584 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:40:57.632427 25584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a: Generated new TSK 0
I20260806 22:40:57.633224 25584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:40:57.693703 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:42951
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:58.041565 25606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:58.041908 25606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:58.042088 25606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:58.049577 25606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:58.049741 25606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:58.049901 25606 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 22:40:58.050043 25606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:40:58.060055 25606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:58.061440 25606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:58.062847 25606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:58.076222 25613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:58.076246 25612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:58.076370 25616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:58.077082 25606 server_base.cc:1034] running on GCE node
I20260806 22:40:58.077769 25606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:58.078755 25606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:58.080014 25606 hybrid_clock.cc:648] HybridClock initialized: now 1786056058079945 us; error 43 us; skew 500 ppm
I20260806 22:40:58.080341 25606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:58 dist-test-slave-zr1t krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056058, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:58.083613 25606 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:40:58.085163 25606 webserver.cc:460] Webserver started at http://127.23.100.129:39947/ using document root <none> and password file <none>
I20260806 22:40:58.085933 25606 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:58.086067 25606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:58.086396 25606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:58.089334 25606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "4f4bedcef96f4a1099d4071d049df07d"
format_stamp: "Formatted at 2026-08-06 22:40:58 on dist-test-slave-zr1t"
server_key: "6556a7348276ee98d2ad9e50c201f392"
server_key_iv: "6b7f1138dd64c73f27fa69de8e09cf78"
server_key_version: "encryptionkey@0"
I20260806 22:40:58.090235 25606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "4f4bedcef96f4a1099d4071d049df07d"
format_stamp: "Formatted at 2026-08-06 22:40:58 on dist-test-slave-zr1t"
server_key: "6556a7348276ee98d2ad9e50c201f392"
server_key_iv: "6b7f1138dd64c73f27fa69de8e09cf78"
server_key_version: "encryptionkey@0"
I20260806 22:40:58.095434 25606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 22:40:58.099309 25625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:58.100580 25606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 22:40:58.100762 25606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "4f4bedcef96f4a1099d4071d049df07d"
format_stamp: "Formatted at 2026-08-06 22:40:58 on dist-test-slave-zr1t"
server_key: "6556a7348276ee98d2ad9e50c201f392"
server_key_iv: "6b7f1138dd64c73f27fa69de8e09cf78"
server_key_version: "encryptionkey@0"
I20260806 22:40:58.101009 25606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:58.118950 25606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:58.123258 25606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:58.123558 25606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:58.126682 25606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:40:58.126820 25606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:58.126959 25606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:40:58.127044 25606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 22:40:58 dist-test-slave-zr1t krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056058, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:58.151000 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:58.127108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:48093 (local address 127.23.100.190:42951)
0806 22:40:58.128079 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:58.128084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:58.128779 (+ 695us) server_negotiation.cc:407] Connection header received
0806 22:40:58.130849 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:58.130856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:58.130955 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:58.131079 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:58.133232 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.134325 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.136777 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.137425 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.137592 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:58.142965 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:58.142974 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:58.142980 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:58.143029 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:58.146705 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:58.147918 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:58.147922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:58.147924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:58.148103 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:58.148813 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:58.148820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:58.148824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:58.149110 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:58.149235 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:58.150543 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0806 22:40:58.150734 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":748,"threads_started":1}
I20260806 22:40:58.153110 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:58.127763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:42951 (local address 127.23.100.129:48093)
0806 22:40:58.128613 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:58.128653 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:58.130561 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:58.131326 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:58.131334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:58.132106 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:58.132972 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:58.133006 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.134494 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.134507 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:58.136568 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:58.136587 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.137708 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.137718 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:58.138069 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:58.138807 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:58.138834 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:58.142651 (+ 3817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:58.146889 (+ 4238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:58.146901 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:58.146922 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:58.147722 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:58.148259 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:58.148270 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:58.148277 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:58.148650 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:58.149264 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:58.149277 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:58.149967 (+ 690us) client_negotiation.cc:769] Sending connection context
0806 22:40:58.150401 (+ 434us) client_negotiation.cc:240] Negotiation successful
0806 22:40:58.151164 (+ 763us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":370,"threads_started":1}
I20260806 22:40:58.174695 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:58.159708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:36245 (local address 127.23.100.190:42951)
0806 22:40:58.159929 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:58.159937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:58.160816 (+ 879us) server_negotiation.cc:407] Connection header received
0806 22:40:58.161040 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:58.161049 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:58.161174 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:58.161322 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:58.162789 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.163819 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.166141 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.166579 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.166730 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:58.168635 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:58.168649 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:58.168653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:58.168686 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:58.171609 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:58.172582 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:58.172589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:58.172592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:58.172735 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:58.173122 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:58.173128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:58.173131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:58.173425 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:58.173620 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:58.174048 (+ 428us) server_negotiation.cc:299] Negotiation successful
0806 22:40:58.174302 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 22:40:58.174817 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:58.159799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:42951 (local address 127.23.100.129:36245)
0806 22:40:58.160668 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:58.160700 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:58.160875 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:58.161330 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:58.161333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:58.161920 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:58.162623 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:58.162636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.163966 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.163972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:58.165984 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:58.165997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.166778 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.166784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:58.166962 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:58.167636 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:58.167654 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:58.168431 (+ 777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:58.171760 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:58.171767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:58.171773 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:58.172441 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:58.172853 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:58.172858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:58.172862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:58.173009 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:58.173568 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:58.173575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:58.173785 (+ 210us) client_negotiation.cc:769] Sending connection context
0806 22:40:58.173957 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 22:40:58.174312 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":490,"threads_started":1}
I20260806 22:40:58.188551 25606 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:43659
I20260806 22:40:58.188575 25799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:43659 every 8 connection(s)
I20260806 22:40:58.189908 25606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:40:58.192137 25800 heartbeater.cc:347] Connected to a master server at 127.23.100.190:42951
I20260806 22:40:58.192394 25800 heartbeater.cc:464] Registering TS with master...
I20260806 22:40:58.193132 25800 heartbeater.cc:511] Master 127.23.100.190:42951 requested a full tablet report, sending...
I20260806 22:40:58.195166 25504 ts_manager.cc:194] Registered new tserver with Master: 4f4bedcef96f4a1099d4071d049df07d (127.23.100.129:43659)
I20260806 22:40:58.196697 25504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:48093
I20260806 22:40:58.199471 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 25606
I20260806 22:40:58.199715 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:40:58.200325 23954 external_mini_cluster.cc:1428] Setting key 4f7c8d1ea85cc4b2f887b47ae82bd9b8
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:40:58.248522 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:42951
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:58.586117 25805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:58.586433 25805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:58.586616 25805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:58.594349 25805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:58.594493 25805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:58.594648 25805 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 22:40:58.594785 25805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:40:58.605201 25805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:58.606722 25805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:58.608106 25805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:58.622044 25816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:58.622171 25813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:58.622124 25814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:58.623208 25805 server_base.cc:1034] running on GCE node
I20260806 22:40:58.623795 25805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:58.624712 25805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:58.625944 25805 hybrid_clock.cc:648] HybridClock initialized: now 1786056058625878 us; error 32 us; skew 500 ppm
I20260806 22:40:58.626267 25805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:58 dist-test-slave-zr1t krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056058, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:58.629688 25805 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:40:58.631178 25805 webserver.cc:460] Webserver started at http://127.23.100.130:37425/ using document root <none> and password file <none>
I20260806 22:40:58.631886 25805 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:58.632011 25805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:58.632368 25805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:58.635318 25805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "dec256cdd30b4020851faee43b550f8b"
format_stamp: "Formatted at 2026-08-06 22:40:58 on dist-test-slave-zr1t"
server_key: "2ddf00d2a38bea5acb2cc7a838b7cf68"
server_key_iv: "de65eac33cfe5cbcfd909418405224b4"
server_key_version: "encryptionkey@0"
I20260806 22:40:58.636130 25805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "dec256cdd30b4020851faee43b550f8b"
format_stamp: "Formatted at 2026-08-06 22:40:58 on dist-test-slave-zr1t"
server_key: "2ddf00d2a38bea5acb2cc7a838b7cf68"
server_key_iv: "de65eac33cfe5cbcfd909418405224b4"
server_key_version: "encryptionkey@0"
I20260806 22:40:58.641268 25805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 22:40:58.644666 25828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:58.646054 25805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:40:58.646225 25805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "dec256cdd30b4020851faee43b550f8b"
format_stamp: "Formatted at 2026-08-06 22:40:58 on dist-test-slave-zr1t"
server_key: "2ddf00d2a38bea5acb2cc7a838b7cf68"
server_key_iv: "de65eac33cfe5cbcfd909418405224b4"
server_key_version: "encryptionkey@0"
I20260806 22:40:58.646476 25805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:58.664963 25805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:58.669646 25805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:58.669984 25805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:58.672915 25805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:40:58.672998 25805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:58.673156 25805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:40:58.673220 25805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 22:40:58 dist-test-slave-zr1t krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056058, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:58.699785 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:58.674322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:42951 (local address 127.23.100.130:45821)
0806 22:40:58.675264 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:58.675323 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:58.678273 (+ 2950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:58.679158 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:58.679166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:58.679702 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:58.680557 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:58.680581 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.682078 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.682087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:58.684885 (+ 2798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:58.684905 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.686207 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.686220 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:58.686604 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:58.687508 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:58.687549 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:58.690946 (+ 3397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:58.695097 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:58.695106 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:58.695127 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:58.695821 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:58.696367 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:58.696372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:58.696375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:58.696655 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:58.697236 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:58.697245 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:58.697741 (+ 496us) client_negotiation.cc:769] Sending connection context
0806 22:40:58.698161 (+ 420us) client_negotiation.cc:240] Negotiation successful
0806 22:40:58.698656 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":351,"threads_started":1}
I20260806 22:40:58.700829 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:58.673748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:45821 (local address 127.23.100.190:42951)
0806 22:40:58.673928 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:58.673933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:58.675458 (+ 1525us) server_negotiation.cc:407] Connection header received
0806 22:40:58.678647 (+ 3189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:58.678654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:58.678756 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:58.678908 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:58.680838 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.681860 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.685187 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.685888 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.686040 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:58.691205 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:58.691215 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:58.691222 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:58.691273 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:58.694927 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:58.696040 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:58.696049 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:58.696055 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:58.696222 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:58.696775 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:58.696782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:58.696785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:58.697089 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:58.697256 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:58.700354 (+ 3098us) server_negotiation.cc:299] Negotiation successful
0806 22:40:58.700575 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 22:40:58.722168 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:58.706650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:42951 (local address 127.23.100.130:57059)
0806 22:40:58.707290 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:58.707307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:58.707482 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:58.707810 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:58.707815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:58.708297 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:58.708596 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:58.708605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.709680 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.709693 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:58.711395 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:58.711413 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.712417 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.712426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:58.712596 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:58.713329 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:58.713362 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:58.714922 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:58.718764 (+ 3842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:58.718769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:58.718774 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:58.719513 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:58.720046 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:58.720053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:58.720057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:58.720187 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:58.721015 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:58.721019 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:58.721182 (+ 163us) client_negotiation.cc:769] Sending connection context
0806 22:40:58.721301 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 22:40:58.721645 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":359,"threads_started":1}
I20260806 22:40:58.722225 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:58.706539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:57059 (local address 127.23.100.190:42951)
0806 22:40:58.706713 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:58.706717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:58.707409 (+ 692us) server_negotiation.cc:407] Connection header received
0806 22:40:58.707614 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:58.707619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:58.707680 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:58.707788 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:58.708764 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.709487 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.711558 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:58.712192 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:58.712331 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:58.715235 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:58.715243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:58.715252 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:58.715297 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:58.718576 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:58.719699 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:58.719707 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:58.719712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:58.719900 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:58.720308 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:58.720317 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:58.720322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:58.720877 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:58.721084 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:58.721449 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 22:40:58.721762 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 22:40:58.730921 25805 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:44063
I20260806 22:40:58.730979 25995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:44063 every 8 connection(s)
I20260806 22:40:58.732156 25805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:40:58.734239 25996 heartbeater.cc:347] Connected to a master server at 127.23.100.190:42951
I20260806 22:40:58.734488 25996 heartbeater.cc:464] Registering TS with master...
I20260806 22:40:58.735132 25996 heartbeater.cc:511] Master 127.23.100.190:42951 requested a full tablet report, sending...
I20260806 22:40:58.736474 25504 ts_manager.cc:194] Registered new tserver with Master: dec256cdd30b4020851faee43b550f8b (127.23.100.130:44063)
I20260806 22:40:58.737267 25504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:45821
I20260806 22:40:58.741496 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 25805
I20260806 22:40:58.741737 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:40:58.742125 23954 external_mini_cluster.cc:1428] Setting key 07f52af889a1c070e106ed82129de542
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:40:58.786882 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:42951
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:40:59.123186 26004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:40:59.123533 26004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:40:59.123713 26004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:40:59.131101 26004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:40:59.131268 26004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:40:59.131430 26004 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 22:40:59.131572 26004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:40:59.141710 26004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.190:42951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:40:59.143107 26004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:40:59.144531 26004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:40:59.158818 26011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:59.158979 26018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:40:59.158818 26012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:40:59.160225 26004 server_base.cc:1034] running on GCE node
I20260806 22:40:59.161007 26004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:40:59.162168 26004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:40:59.163419 26004 hybrid_clock.cc:648] HybridClock initialized: now 1786056059163317 us; error 75 us; skew 500 ppm
I20260806 22:40:59.163760 26004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:40:59 dist-test-slave-zr1t krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056059, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:40:59.174679 26004 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:40:59.176569 26004 webserver.cc:460] Webserver started at http://127.23.100.131:43923/ using document root <none> and password file <none>
I20260806 22:40:59.177449 26004 fs_manager.cc:359] Metadata directory not provided
I20260806 22:40:59.177632 26004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:40:59.178040 26004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:40:59.181356 26004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "d75c199400b54951ac76898cd6db4b53"
format_stamp: "Formatted at 2026-08-06 22:40:59 on dist-test-slave-zr1t"
server_key: "f067f990d04acd2fd5cbe91bdc998d4a"
server_key_iv: "42648e7bef7d2dff533ac02cb8b8e840"
server_key_version: "encryptionkey@0"
I20260806 22:40:59.182407 26004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "d75c199400b54951ac76898cd6db4b53"
format_stamp: "Formatted at 2026-08-06 22:40:59 on dist-test-slave-zr1t"
server_key: "f067f990d04acd2fd5cbe91bdc998d4a"
server_key_iv: "42648e7bef7d2dff533ac02cb8b8e840"
server_key_version: "encryptionkey@0"
I20260806 22:40:59.188376 26004 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 22:40:59.192521 26025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:40:59.194183 26004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 22:40:59.194360 26004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "d75c199400b54951ac76898cd6db4b53"
format_stamp: "Formatted at 2026-08-06 22:40:59 on dist-test-slave-zr1t"
server_key: "f067f990d04acd2fd5cbe91bdc998d4a"
server_key_iv: "42648e7bef7d2dff533ac02cb8b8e840"
server_key_version: "encryptionkey@0"
I20260806 22:40:59.194625 26004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:40:59.199955 25800 heartbeater.cc:503] Master 127.23.100.190:42951 was elected leader, sending a full tablet report...
I20260806 22:40:59.210706 26004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:40:59.215312 26004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:40:59.215595 26004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:40:59.218595 26004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:40:59.218729 26004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 22:40:59.218902 26004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:40:59.218997 26004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 22:40:59 dist-test-slave-zr1t krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056059, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:59.244082 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.219162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:48823 (local address 127.23.100.190:42951)
0806 22:40:59.219378 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.219384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.220886 (+ 1502us) server_negotiation.cc:407] Connection header received
0806 22:40:59.223228 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.223234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.223328 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.223456 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:59.225847 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.227245 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.228979 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.229849 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.230022 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.235684 (+ 5662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:59.235689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:59.235693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:59.235729 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:59.238834 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:59.240126 (+ 1292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:59.240134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:59.240137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:59.240312 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:59.240979 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:59.240987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:59.240993 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:59.241444 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:59.241686 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.243131 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.243527 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 22:40:59.244534 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.219731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:42951 (local address 127.23.100.131:48823)
0806 22:40:59.220672 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.220730 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.222874 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.223723 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.223737 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.224629 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:59.225573 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.225608 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.227459 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.227467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.228820 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.228834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.230126 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.230135 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.230416 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.231360 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:59.231405 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:59.235394 (+ 3989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:59.239015 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:59.239033 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:59.239061 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:59.239940 (+ 879us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:59.240440 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:59.240446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:59.240450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:59.240804 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:59.241633 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:59.241645 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:59.242153 (+ 508us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.242664 (+ 511us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.243261 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":386,"threads_started":1}
I20260806 22:40:59.268409 26004 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:43785
I20260806 22:40:59.268451 26194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:43785 every 8 connection(s)
I20260806 22:40:59.269804 26004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:40:59.271955 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.254041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:45289 (local address 127.23.100.190:42951)
0806 22:40:59.254254 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.254263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.255083 (+ 820us) server_negotiation.cc:407] Connection header received
0806 22:40:59.255425 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.255434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.255532 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.255624 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:59.257569 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.258804 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.261135 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.261757 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.261900 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.264859 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:59.264864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:59.264867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:59.264900 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:59.268555 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:59.269259 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:59.269266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:59.269270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:59.269408 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:59.269864 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:59.269869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:59.269873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:59.270179 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:59.270362 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.271293 (+ 931us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.271549 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:40:59.272054 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.254111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:42951 (local address 127.23.100.131:45289)
0806 22:40:59.254946 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.254973 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.255223 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.255886 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.255891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.256747 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:59.257330 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.257347 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.258983 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.258988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.260948 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.260971 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.262131 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.262136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.262439 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.263215 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:59.263245 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:59.264607 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:59.268720 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:59.268724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:59.268726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:59.269127 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:59.269582 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:59.269586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:59.269588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:59.269709 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:59.270388 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:59.270393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:59.270674 (+ 281us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.271191 (+ 517us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.271549 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":449,"threads_started":1}
I20260806 22:40:59.272141 26195 heartbeater.cc:347] Connected to a master server at 127.23.100.190:42951
I20260806 22:40:59.272902 26195 heartbeater.cc:464] Registering TS with master...
I20260806 22:40:59.273782 26195 heartbeater.cc:511] Master 127.23.100.190:42951 requested a full tablet report, sending...
I20260806 22:40:59.275215 25504 ts_manager.cc:194] Registered new tserver with Master: d75c199400b54951ac76898cd6db4b53 (127.23.100.131:43785)
I20260806 22:40:59.276239 25504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:48823
I20260806 22:40:59.278630 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 26004
I20260806 22:40:59.278807 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:40:59.279186 23954 external_mini_cluster.cc:1428] Setting key da4dd3bafa60e705ffe1c331f6b3a760
I20260806 22:40:59.282121 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 22:40:59 dist-test-slave-zr1t krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 22:40:59 dist-test-slave-zr1t krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:59.336220 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.322613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.23.100.190:42951)
0806 22:40:59.322870 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.322878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.323093 (+ 215us) server_negotiation.cc:407] Connection header received
0806 22:40:59.323241 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.323245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.323314 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.323428 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:59.324770 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.325619 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.326854 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.327270 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.327364 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.331331 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:59.331337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:59.331340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:59.331401 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:59.333943 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:59.334519 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:59.334522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:59.334524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:59.334643 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:59.334959 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:59.334962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:59.334964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:59.335221 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:59.335423 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.335663 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.335829 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 22:40:59.340476 25504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:40:59.347452 25504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:40:59.383303 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.360269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.23.100.130:44063)
0806 22:40:59.374627 (+ 14358us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.374640 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.374679 (+ 39us) server_negotiation.cc:407] Connection header received
0806 22:40:59.374800 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.374806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.375134 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.375337 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:59.376737 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.378181 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.380895 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.382126 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.382345 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.382548 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.382811 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.382973 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14130,"thread_start_us":13958,"threads_started":1}
I20260806 22:40:59.383440 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.359549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:44063 (local address 127.0.0.1:46856)
0806 22:40:59.373283 (+ 13734us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.373353 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.373607 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.376008 (+ 2401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.376021 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.376053 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:59.376522 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.376540 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.378325 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.378332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.380699 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.380714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.382346 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.382352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.382542 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.382597 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.382769 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.382973 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13384,"mutex_wait_us":75,"thread_start_us":15231,"threads_started":1}
I20260806 22:40:59.383633 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.359791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43659 (local address 127.0.0.1:58596)
0806 22:40:59.375156 (+ 15365us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.375180 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.375432 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.376336 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.376340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.376369 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:59.376892 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.376904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.378457 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.378462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.380588 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.380603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.382748 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.382755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.382922 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.382955 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.383048 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.383190 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15181,"mutex_wait_us":56,"thread_start_us":15894,"threads_started":1}
I20260806 22:40:59.384348 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.360901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.23.100.129:43659)
0806 22:40:59.374582 (+ 13681us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.374590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.375303 (+ 713us) server_negotiation.cc:407] Connection header received
0806 22:40:59.375657 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.375668 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.375900 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.376063 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:59.377093 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.378303 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.380788 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.382227 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.382418 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.382587 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.383860 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.384039 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13511,"thread_start_us":13523,"threads_started":1}
I20260806 22:40:59.384449 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.360374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:43785 (local address 127.0.0.1:49240)
0806 22:40:59.375944 (+ 15570us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.375962 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.376184 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.376806 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.376813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.376845 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:59.377240 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.377253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.379256 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.379261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.381467 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.381482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.383607 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.383619 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.383823 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.383854 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.383945 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.384076 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15458,"mutex_wait_us":3,"spinlock_wait_cycles":3456,"thread_start_us":12357,"threads_started":1}
I20260806 22:40:59.385133 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.361163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.23.100.131:43785)
0806 22:40:59.374137 (+ 12974us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.374149 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.376054 (+ 1905us) server_negotiation.cc:407] Connection header received
0806 22:40:59.376413 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.376419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.376561 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.376794 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:59.377468 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.379108 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.381657 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.383112 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.383283 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.383446 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.384583 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.384770 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12761,"thread_start_us":12630,"threads_started":1}
I20260806 22:40:59.387820 25702 tablet_service.cc:1467] Processing CreateTablet for tablet a4a7efed474642ceb44ec974e4387381 (DEFAULT_TABLE table=test-table [id=cb7ea93e675442efb5542d8ec03e0b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:40:59.387778 25899 tablet_service.cc:1467] Processing CreateTablet for tablet a4a7efed474642ceb44ec974e4387381 (DEFAULT_TABLE table=test-table [id=cb7ea93e675442efb5542d8ec03e0b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:40:59.388684 26096 tablet_service.cc:1467] Processing CreateTablet for tablet a4a7efed474642ceb44ec974e4387381 (DEFAULT_TABLE table=test-table [id=cb7ea93e675442efb5542d8ec03e0b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:40:59.389202 25899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a7efed474642ceb44ec974e4387381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:59.389187 25702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a7efed474642ceb44ec974e4387381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:59.390051 26096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a7efed474642ceb44ec974e4387381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:59.400852 26220 tablet_bootstrap.cc:492] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: Bootstrap starting.
I20260806 22:40:59.404083 26221 tablet_bootstrap.cc:492] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: Bootstrap starting.
I20260806 22:40:59.405861 26223 tablet_bootstrap.cc:492] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: Bootstrap starting.
I20260806 22:40:59.406234 26220 tablet_bootstrap.cc:654] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:59.407639 26220 log.cc:824] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: Log is configured to *not* fsync() on all Append() calls
I20260806 22:40:59.407922 26221 tablet_bootstrap.cc:654] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:59.409029 26221 log.cc:824] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: Log is configured to *not* fsync() on all Append() calls
I20260806 22:40:59.409075 26223 tablet_bootstrap.cc:654] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:59.410439 26223 log.cc:824] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: Log is configured to *not* fsync() on all Append() calls
I20260806 22:40:59.410764 26220 tablet_bootstrap.cc:492] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: No bootstrap required, opened a new log
I20260806 22:40:59.411051 26220 ts_tablet_manager.cc:1397] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 22:40:59.412084 26221 tablet_bootstrap.cc:492] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: No bootstrap required, opened a new log
I20260806 22:40:59.412357 26221 ts_tablet_manager.cc:1397] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260806 22:40:59.412947 26223 tablet_bootstrap.cc:492] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: No bootstrap required, opened a new log
I20260806 22:40:59.413211 26223 ts_tablet_manager.cc:1397] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 22:40:59.421446 26223 raft_consensus.cc:348] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.421927 26223 raft_consensus.cc:374] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:59.422096 26223 raft_consensus.cc:729] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75c199400b54951ac76898cd6db4b53, State: Initialized, Role: FOLLOWER
I20260806 22:40:59.422721 26223 consensus_queue.cc:260] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.422770 26220 raft_consensus.cc:348] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.423287 26220 raft_consensus.cc:374] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:59.423095 26221 raft_consensus.cc:348] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.423426 26220 raft_consensus.cc:729] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4bedcef96f4a1099d4071d049df07d, State: Initialized, Role: FOLLOWER
I20260806 22:40:59.423561 26221 raft_consensus.cc:374] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:59.423703 26221 raft_consensus.cc:729] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec256cdd30b4020851faee43b550f8b, State: Initialized, Role: FOLLOWER
I20260806 22:40:59.423867 26195 heartbeater.cc:503] Master 127.23.100.190:42951 was elected leader, sending a full tablet report...
I20260806 22:40:59.424082 26220 consensus_queue.cc:260] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.424389 26221 consensus_queue.cc:260] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.426119 26220 ts_tablet_manager.cc:1428] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260806 22:40:59.427089 25996 heartbeater.cc:503] Master 127.23.100.190:42951 was elected leader, sending a full tablet report...
I20260806 22:40:59.427824 26221 ts_tablet_manager.cc:1428] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260806 22:40:59.428498 26223 ts_tablet_manager.cc:1428] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260806 22:40:59.442364 25801 tablet.cc:2369] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:40:59.477942 26232 raft_consensus.cc:482] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:40:59.478312 26232 raft_consensus.cc:504] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.480046 26232 leader_election.cc:288] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f4bedcef96f4a1099d4071d049df07d (127.23.100.129:43659), d75c199400b54951ac76898cd6db4b53 (127.23.100.131:43785)
W20260806 22:40:59.484273 25998 tablet.cc:2369] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:40:59.487804 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.480619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:42193 (local address 127.23.100.131:43785)
0806 22:40:59.480861 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.480870 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.481558 (+ 688us) server_negotiation.cc:407] Connection header received
0806 22:40:59.481788 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.481794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.481867 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.481935 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:59.482725 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.483547 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.485826 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.486758 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.486948 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.487006 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.487390 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.487546 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 22:40:59.488121 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.480419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:43785 (local address 127.23.100.130:42193)
0806 22:40:59.481403 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.481420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.481620 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.481994 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.482003 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.482030 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:59.482525 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.482536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.483698 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.483707 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.485667 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.485680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.487020 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.487027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.487212 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.487253 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.487350 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.487501 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":118,"thread_start_us":569,"threads_started":1}
I20260806 22:40:59.488211 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.480489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:53241 (local address 127.23.100.129:43659)
0806 22:40:59.480732 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.480741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.481615 (+ 874us) server_negotiation.cc:407] Connection header received
0806 22:40:59.481833 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.481837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.481902 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.482021 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:59.482927 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.483660 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.486239 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.487213 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.487398 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.487471 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.487825 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.487964 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 22:40:59.488528 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.480510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43659 (local address 127.23.100.130:53241)
0806 22:40:59.481467 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.481506 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.481666 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.482058 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.482063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.482095 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:59.482685 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.482703 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.483805 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.483809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.486080 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.486095 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.487455 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.487467 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.487639 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.487674 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.487775 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.487943 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":51,"thread_start_us":582,"threads_started":1}
I20260806 22:40:59.489389 26124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a7efed474642ceb44ec974e4387381" candidate_uuid: "dec256cdd30b4020851faee43b550f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d75c199400b54951ac76898cd6db4b53" is_pre_election: true
I20260806 22:40:59.489670 25736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a7efed474642ceb44ec974e4387381" candidate_uuid: "dec256cdd30b4020851faee43b550f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4bedcef96f4a1099d4071d049df07d" is_pre_election: true
I20260806 22:40:59.490008 26124 raft_consensus.cc:2454] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec256cdd30b4020851faee43b550f8b in term 0.
I20260806 22:40:59.490267 25736 raft_consensus.cc:2454] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec256cdd30b4020851faee43b550f8b in term 0.
I20260806 22:40:59.490775 25830 leader_election.cc:302] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d75c199400b54951ac76898cd6db4b53, dec256cdd30b4020851faee43b550f8b; no voters:
I20260806 22:40:59.491271 26232 raft_consensus.cc:2781] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:40:59.491454 26232 raft_consensus.cc:482] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:40:59.491572 26232 raft_consensus.cc:3037] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:59.493945 26232 raft_consensus.cc:504] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.494551 26232 leader_election.cc:288] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [CANDIDATE]: Term 1 election: Requested vote from peers 4f4bedcef96f4a1099d4071d049df07d (127.23.100.129:43659), d75c199400b54951ac76898cd6db4b53 (127.23.100.131:43785)
I20260806 22:40:59.495090 25736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a7efed474642ceb44ec974e4387381" candidate_uuid: "dec256cdd30b4020851faee43b550f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4bedcef96f4a1099d4071d049df07d"
I20260806 22:40:59.495090 26124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a7efed474642ceb44ec974e4387381" candidate_uuid: "dec256cdd30b4020851faee43b550f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d75c199400b54951ac76898cd6db4b53"
I20260806 22:40:59.495307 25736 raft_consensus.cc:3037] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:59.495307 26124 raft_consensus.cc:3037] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:59.498677 26124 raft_consensus.cc:2454] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec256cdd30b4020851faee43b550f8b in term 1.
I20260806 22:40:59.498677 25736 raft_consensus.cc:2454] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec256cdd30b4020851faee43b550f8b in term 1.
I20260806 22:40:59.499161 25829 leader_election.cc:302] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f4bedcef96f4a1099d4071d049df07d, d75c199400b54951ac76898cd6db4b53, dec256cdd30b4020851faee43b550f8b; no voters:
I20260806 22:40:59.499523 26232 raft_consensus.cc:2781] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:40:59.499989 26232 raft_consensus.cc:686] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 1 LEADER]: Becoming Leader. State: Replica: dec256cdd30b4020851faee43b550f8b, State: Running, Role: LEADER
I20260806 22:40:59.500536 26232 consensus_queue.cc:237] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.505129 25503 catalog_manager.cc:5495] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to dec256cdd30b4020851faee43b550f8b (127.23.100.130). New cstate: current_term: 1 leader_uuid: "dec256cdd30b4020851faee43b550f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } health_report { overall_health: UNKNOWN } } }
W20260806 22:40:59.521718 26196 tablet.cc:2369] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:40:59.539870 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.534266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.23.100.130:44063)
0806 22:40:59.534531 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.534537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.534559 (+ 22us) server_negotiation.cc:407] Connection header received
0806 22:40:59.534809 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.534813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.534894 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.535048 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:40:59.535445 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.536179 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.537599 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.538031 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.538174 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.538685 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:40:59.539200 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:40:59.539367 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.539506 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.539591 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 22:40:59.548306 25736 raft_consensus.cc:1261] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d [term 1 FOLLOWER]: Refusing update from remote peer dec256cdd30b4020851faee43b550f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:40:59.548316 26124 raft_consensus.cc:1261] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53 [term 1 FOLLOWER]: Refusing update from remote peer dec256cdd30b4020851faee43b550f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:40:59.549481 26237 consensus_queue.cc:1035] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:40:59.549981 26232 consensus_queue.cc:1035] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:40:59.588583 25503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:40:59.588917 25503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:40:59.593314 25503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a: Sending DeleteTablet for 3 replicas of tablet a4a7efed474642ceb44ec974e4387381
I20260806 22:40:59.594719 25899 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a7efed474642ceb44ec974e4387381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:40:59 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:46856
I20260806 22:40:59.594784 25702 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a7efed474642ceb44ec974e4387381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:40:59 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:58596
I20260806 22:40:59.595645 26096 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a7efed474642ceb44ec974e4387381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:40:59 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:49240
I20260806 22:40:59.596467 25503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:40:59.597260 25503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:40:59.599943 26257 tablet_replica.cc:330] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: stopping tablet replica
I20260806 22:40:59.600497 26257 raft_consensus.cc:2229] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:40:59.601027 26257 raft_consensus.cc:2258] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:40:59.601866 26253 tablet_replica.cc:330] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: stopping tablet replica
I20260806 22:40:59.602581 26253 raft_consensus.cc:2229] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:40:59.603132 26253 raft_consensus.cc:2258] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:40:59.604156 26252 tablet_replica.cc:330] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: stopping tablet replica
I20260806 22:40:59.604672 26252 raft_consensus.cc:2229] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:40:59.605474 26252 raft_consensus.cc:2258] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:40:59.607995 26257 ts_tablet_manager.cc:1905] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:40:59.609711 25702 tablet_service.cc:1467] Processing CreateTablet for tablet c0998f4a56be4577a08775852ab02a89 (DEFAULT_TABLE table=test-table [id=57e8956403454b028c7554ea74575bc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:40:59.609999 26096 tablet_service.cc:1467] Processing CreateTablet for tablet c0998f4a56be4577a08775852ab02a89 (DEFAULT_TABLE table=test-table [id=57e8956403454b028c7554ea74575bc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:40:59.610271 25702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0998f4a56be4577a08775852ab02a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:59.610473 26096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0998f4a56be4577a08775852ab02a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:59.611526 26253 ts_tablet_manager.cc:1905] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:40:59.619268 25899 tablet_service.cc:1467] Processing CreateTablet for tablet c0998f4a56be4577a08775852ab02a89 (DEFAULT_TABLE table=test-table [id=57e8956403454b028c7554ea74575bc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:40:59.620205 25899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0998f4a56be4577a08775852ab02a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:59.620842 26223 tablet_bootstrap.cc:492] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53: Bootstrap starting.
I20260806 22:40:59.621126 26253 ts_tablet_manager.cc:1918] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:40:59.621325 26253 log.cc:1196] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/a4a7efed474642ceb44ec974e4387381
I20260806 22:40:59.621987 26253 ts_tablet_manager.cc:1939] T a4a7efed474642ceb44ec974e4387381 P 4f4bedcef96f4a1099d4071d049df07d: Deleting consensus metadata
I20260806 22:40:59.622783 26223 tablet_bootstrap.cc:654] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:59.624104 25481 catalog_manager.cc:4841] TS 4f4bedcef96f4a1099d4071d049df07d (127.23.100.129:43659): tablet a4a7efed474642ceb44ec974e4387381 (table test-table [id=cb7ea93e675442efb5542d8ec03e0b63]) successfully deleted
I20260806 22:40:59.627039 26257 ts_tablet_manager.cc:1918] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:40:59.627239 26257 log.cc:1196] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/a4a7efed474642ceb44ec974e4387381
I20260806 22:40:59.627986 26257 ts_tablet_manager.cc:1939] T a4a7efed474642ceb44ec974e4387381 P d75c199400b54951ac76898cd6db4b53: Deleting consensus metadata
I20260806 22:40:59.630093 25482 catalog_manager.cc:4841] TS d75c199400b54951ac76898cd6db4b53 (127.23.100.131:43785): tablet a4a7efed474642ceb44ec974e4387381 (table test-table [id=cb7ea93e675442efb5542d8ec03e0b63]) successfully deleted
I20260806 22:40:59.630589 26252 ts_tablet_manager.cc:1905] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:40:59.630914 26220 tablet_bootstrap.cc:492] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d: Bootstrap starting.
I20260806 22:40:59.632899 26220 tablet_bootstrap.cc:654] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:59.637213 26221 tablet_bootstrap.cc:492] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b: Bootstrap starting.
I20260806 22:40:59.639840 26221 tablet_bootstrap.cc:654] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:59.640672 26252 ts_tablet_manager.cc:1918] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:40:59.640894 26252 log.cc:1196] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/a4a7efed474642ceb44ec974e4387381
I20260806 22:40:59.642520 26252 ts_tablet_manager.cc:1939] T a4a7efed474642ceb44ec974e4387381 P dec256cdd30b4020851faee43b550f8b: Deleting consensus metadata
I20260806 22:40:59.643265 26223 tablet_bootstrap.cc:492] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53: No bootstrap required, opened a new log
I20260806 22:40:59.643503 26223 ts_tablet_manager.cc:1397] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.013s
I20260806 22:40:59.644583 25483 catalog_manager.cc:4841] TS dec256cdd30b4020851faee43b550f8b (127.23.100.130:44063): tablet a4a7efed474642ceb44ec974e4387381 (table test-table [id=cb7ea93e675442efb5542d8ec03e0b63]) successfully deleted
I20260806 22:40:59.644495 26223 raft_consensus.cc:348] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.644953 26223 raft_consensus.cc:374] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:59.645120 26223 raft_consensus.cc:729] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75c199400b54951ac76898cd6db4b53, State: Initialized, Role: FOLLOWER
I20260806 22:40:59.645579 26223 consensus_queue.cc:260] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.646075 26220 tablet_bootstrap.cc:492] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d: No bootstrap required, opened a new log
I20260806 22:40:59.646247 26220 ts_tablet_manager.cc:1397] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260806 22:40:59.646627 26223 ts_tablet_manager.cc:1428] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 22:40:59.647285 26220 raft_consensus.cc:348] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.647611 26220 raft_consensus.cc:374] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:59.647729 26220 raft_consensus.cc:729] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4bedcef96f4a1099d4071d049df07d, State: Initialized, Role: FOLLOWER
I20260806 22:40:59.648056 26220 consensus_queue.cc:260] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.649643 26220 ts_tablet_manager.cc:1428] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 22:40:59.650089 26221 tablet_bootstrap.cc:492] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b: No bootstrap required, opened a new log
I20260806 22:40:59.650315 26221 ts_tablet_manager.cc:1397] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260806 22:40:59.651472 26221 raft_consensus.cc:348] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.651902 26221 raft_consensus.cc:374] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:59.652055 26221 raft_consensus.cc:729] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec256cdd30b4020851faee43b550f8b, State: Initialized, Role: FOLLOWER
I20260806 22:40:59.652356 26221 consensus_queue.cc:260] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.652944 26221 ts_tablet_manager.cc:1428] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 22:40:59.681785 26245 raft_consensus.cc:482] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:40:59.682035 26245 raft_consensus.cc:504] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.682914 26245 leader_election.cc:288] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f4bedcef96f4a1099d4071d049df07d (127.23.100.129:43659), d75c199400b54951ac76898cd6db4b53 (127.23.100.131:43785)
I20260806 22:40:59.683481 25736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0998f4a56be4577a08775852ab02a89" candidate_uuid: "dec256cdd30b4020851faee43b550f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4bedcef96f4a1099d4071d049df07d" is_pre_election: true
I20260806 22:40:59.683504 26124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0998f4a56be4577a08775852ab02a89" candidate_uuid: "dec256cdd30b4020851faee43b550f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d75c199400b54951ac76898cd6db4b53" is_pre_election: true
I20260806 22:40:59.683743 25736 raft_consensus.cc:2454] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec256cdd30b4020851faee43b550f8b in term 0.
I20260806 22:40:59.683801 26124 raft_consensus.cc:2454] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dec256cdd30b4020851faee43b550f8b in term 0.
I20260806 22:40:59.684207 25829 leader_election.cc:302] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f4bedcef96f4a1099d4071d049df07d, dec256cdd30b4020851faee43b550f8b; no voters:
I20260806 22:40:59.684492 26245 raft_consensus.cc:2781] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:40:59.684635 26245 raft_consensus.cc:482] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:40:59.684751 26245 raft_consensus.cc:3037] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:59.687183 26245 raft_consensus.cc:504] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.687752 26245 leader_election.cc:288] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [CANDIDATE]: Term 1 election: Requested vote from peers 4f4bedcef96f4a1099d4071d049df07d (127.23.100.129:43659), d75c199400b54951ac76898cd6db4b53 (127.23.100.131:43785)
I20260806 22:40:59.688266 26124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0998f4a56be4577a08775852ab02a89" candidate_uuid: "dec256cdd30b4020851faee43b550f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d75c199400b54951ac76898cd6db4b53"
I20260806 22:40:59.688277 25736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0998f4a56be4577a08775852ab02a89" candidate_uuid: "dec256cdd30b4020851faee43b550f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4bedcef96f4a1099d4071d049df07d"
I20260806 22:40:59.688472 26124 raft_consensus.cc:3037] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:59.688509 25736 raft_consensus.cc:3037] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:59.690629 26124 raft_consensus.cc:2454] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec256cdd30b4020851faee43b550f8b in term 1.
I20260806 22:40:59.691061 25830 leader_election.cc:302] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d75c199400b54951ac76898cd6db4b53, dec256cdd30b4020851faee43b550f8b; no voters:
I20260806 22:40:59.691366 26245 raft_consensus.cc:2781] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:40:59.691581 25736 raft_consensus.cc:2454] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dec256cdd30b4020851faee43b550f8b in term 1.
I20260806 22:40:59.691753 26245 raft_consensus.cc:686] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 1 LEADER]: Becoming Leader. State: Replica: dec256cdd30b4020851faee43b550f8b, State: Running, Role: LEADER
I20260806 22:40:59.692133 26245 consensus_queue.cc:237] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.695076 25503 catalog_manager.cc:5495] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to dec256cdd30b4020851faee43b550f8b (127.23.100.130). New cstate: current_term: 1 leader_uuid: "dec256cdd30b4020851faee43b550f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } health_report { overall_health: UNKNOWN } } }
Aug 06 22:40:59 dist-test-slave-zr1t krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056057, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:40:59.764569 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.744503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.23.100.190:42951)
0806 22:40:59.744728 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.744738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.745842 (+ 1104us) server_negotiation.cc:407] Connection header received
0806 22:40:59.746049 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.746055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.746143 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.746263 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:40:59.747966 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.749225 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.751425 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.752121 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.752258 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.757310 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:40:59.757317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:40:59.757323 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:40:59.757364 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:40:59.761187 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:59.762166 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:59.762173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:59.762177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:59.762346 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:40:59.762799 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:40:59.762804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:40:59.762808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:40:59.763211 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:40:59.763385 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.763899 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.764280 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":72}
I20260806 22:40:59.764593 26269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.744557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:42951 (local address 127.0.0.1:49700)
0806 22:40:59.745683 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.745714 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.745907 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.746312 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.746318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.747342 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:40:59.747740 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.747759 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.749400 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.749406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.751266 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.751283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.752342 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.752349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.752549 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.753723 (+ 1174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:40:59.753755 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:40:59.757069 (+ 3314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:40:59.761342 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:59.761356 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:59.761366 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:59.762021 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:59.762472 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:40:59.762478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:40:59.762483 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:40:59.762666 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:40:59.763363 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:40:59.763372 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:40:59.763595 (+ 223us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.763786 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.764104 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":424,"threads_started":1}
I20260806 22:40:59.770175 25503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:49700:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 22:40:59.771309 25503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:40:59.778520 25702 tablet_service.cc:1467] Processing CreateTablet for tablet b01bd411326d4bd89630229c243bd4f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=30cb0a7fc424498cb57aa6ed08cbe0c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 22:40:59.779163 25702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01bd411326d4bd89630229c243bd4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:59.779117 25899 tablet_service.cc:1467] Processing CreateTablet for tablet b01bd411326d4bd89630229c243bd4f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=30cb0a7fc424498cb57aa6ed08cbe0c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 22:40:59.779350 26096 tablet_service.cc:1467] Processing CreateTablet for tablet b01bd411326d4bd89630229c243bd4f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=30cb0a7fc424498cb57aa6ed08cbe0c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 22:40:59.779717 25899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01bd411326d4bd89630229c243bd4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:59.779856 26096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01bd411326d4bd89630229c243bd4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:40:59.788552 26221 tablet_bootstrap.cc:492] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b: Bootstrap starting.
I20260806 22:40:59.788564 26220 tablet_bootstrap.cc:492] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d: Bootstrap starting.
I20260806 22:40:59.790524 26220 tablet_bootstrap.cc:654] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:59.790537 26221 tablet_bootstrap.cc:654] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:59.790532 26223 tablet_bootstrap.cc:492] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: Bootstrap starting.
I20260806 22:40:59.792596 26223 tablet_bootstrap.cc:654] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: Neither blocks nor log segments found. Creating new log.
I20260806 22:40:59.799567 26223 tablet_bootstrap.cc:492] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: No bootstrap required, opened a new log
I20260806 22:40:59.799763 26223 ts_tablet_manager.cc:1397] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260806 22:40:59.800094 26220 tablet_bootstrap.cc:492] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d: No bootstrap required, opened a new log
I20260806 22:40:59.800272 26220 ts_tablet_manager.cc:1397] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260806 22:40:59.800662 26223 raft_consensus.cc:348] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.800954 26223 raft_consensus.cc:374] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:59.801041 26223 raft_consensus.cc:729] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75c199400b54951ac76898cd6db4b53, State: Initialized, Role: FOLLOWER
I20260806 22:40:59.801165 26220 raft_consensus.cc:348] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.801438 26220 raft_consensus.cc:374] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:59.801311 26223 consensus_queue.cc:260] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.801538 26220 raft_consensus.cc:729] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4bedcef96f4a1099d4071d049df07d, State: Initialized, Role: FOLLOWER
I20260806 22:40:59.801579 26221 tablet_bootstrap.cc:492] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b: No bootstrap required, opened a new log
I20260806 22:40:59.801791 26221 ts_tablet_manager.cc:1397] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260806 22:40:59.801988 26223 ts_tablet_manager.cc:1428] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 22:40:59.801882 26220 consensus_queue.cc:260] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.802513 26231 tablet_replica.cc:439] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } } }
I20260806 22:40:59.802860 26231 tablet_replica.cc:442] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:40:59.802775 26221 raft_consensus.cc:348] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.803069 26221 raft_consensus.cc:374] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:40:59.803153 26221 raft_consensus.cc:729] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec256cdd30b4020851faee43b550f8b, State: Initialized, Role: FOLLOWER
I20260806 22:40:59.803424 26220 ts_tablet_manager.cc:1428] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 22:40:59.804065 26230 tablet_replica.cc:439] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } } }
I20260806 22:40:59.804385 26230 tablet_replica.cc:442] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:40:59.805305 26221 consensus_queue.cc:260] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.806613 26221 ts_tablet_manager.cc:1428] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260806 22:40:59.806627 26245 tablet_replica.cc:439] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } } }
I20260806 22:40:59.807672 26245 tablet_replica.cc:442] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:40:59.828763 26230 raft_consensus.cc:482] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:40:59.829027 26230 raft_consensus.cc:504] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.830991 26230 leader_election.cc:288] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f4bedcef96f4a1099d4071d049df07d (127.23.100.129:43659), dec256cdd30b4020851faee43b550f8b (127.23.100.130:44063)
I20260806 22:40:59.841202 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.831434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:39889 (local address 127.23.100.129:43659)
0806 22:40:59.831701 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.831710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.834305 (+ 2595us) server_negotiation.cc:407] Connection header received
0806 22:40:59.834647 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.834655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.834741 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.834831 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:59.836001 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.837116 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.839138 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.840071 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.840256 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.840356 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.840756 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.840899 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260806 22:40:59.841670 26275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.831221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43659 (local address 127.23.100.131:39889)
0806 22:40:59.834166 (+ 2945us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.834190 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.834451 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.835310 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.835316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.835345 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:59.835796 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.835811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.837252 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.837258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.838956 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.838970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.840279 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.840287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.840525 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.840568 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.840686 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.840850 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2769,"mutex_wait_us":83,"thread_start_us":2993,"threads_started":1}
I20260806 22:40:59.842582 26274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.831218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:44063 (local address 127.23.100.131:33387)
0806 22:40:59.834586 (+ 3368us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.834606 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.834731 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.835184 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.835188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.835212 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:40:59.835488 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.835500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.837091 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.837097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.839695 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.839713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.841502 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.841510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.841702 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.841725 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.842047 (+ 322us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.842189 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3221,"mutex_wait_us":585,"thread_start_us":2022,"threads_started":1}
I20260806 22:40:59.842571 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.831472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:33387 (local address 127.23.100.130:44063)
0806 22:40:59.831644 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.831650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.834716 (+ 3066us) server_negotiation.cc:407] Connection header received
0806 22:40:59.834881 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.834887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.834992 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.835101 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:40:59.835675 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.836919 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.839901 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.841078 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.841256 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.841332 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.842096 (+ 764us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.842232 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 22:40:59.842489 25736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01bd411326d4bd89630229c243bd4f4" candidate_uuid: "d75c199400b54951ac76898cd6db4b53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4bedcef96f4a1099d4071d049df07d" is_pre_election: true
I20260806 22:40:59.842789 25736 raft_consensus.cc:2454] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d75c199400b54951ac76898cd6db4b53 in term 0.
I20260806 22:40:59.843407 26026 leader_election.cc:302] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f4bedcef96f4a1099d4071d049df07d, d75c199400b54951ac76898cd6db4b53; no voters:
I20260806 22:40:59.843628 25938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01bd411326d4bd89630229c243bd4f4" candidate_uuid: "d75c199400b54951ac76898cd6db4b53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec256cdd30b4020851faee43b550f8b" is_pre_election: true
I20260806 22:40:59.843875 26230 raft_consensus.cc:2781] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:40:59.843991 26230 raft_consensus.cc:482] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:40:59.844084 26230 raft_consensus.cc:3037] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:59.844120 25938 raft_consensus.cc:2454] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d75c199400b54951ac76898cd6db4b53 in term 0.
I20260806 22:40:59.846638 26230 raft_consensus.cc:504] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.847373 26230 leader_election.cc:288] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [CANDIDATE]: Term 1 election: Requested vote from peers 4f4bedcef96f4a1099d4071d049df07d (127.23.100.129:43659), dec256cdd30b4020851faee43b550f8b (127.23.100.130:44063)
I20260806 22:40:59.847908 25736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01bd411326d4bd89630229c243bd4f4" candidate_uuid: "d75c199400b54951ac76898cd6db4b53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4bedcef96f4a1099d4071d049df07d"
I20260806 22:40:59.848102 25938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01bd411326d4bd89630229c243bd4f4" candidate_uuid: "d75c199400b54951ac76898cd6db4b53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec256cdd30b4020851faee43b550f8b"
I20260806 22:40:59.848172 25736 raft_consensus.cc:3037] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:59.848328 25938 raft_consensus.cc:3037] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:40:59.850890 25938 raft_consensus.cc:2454] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d75c199400b54951ac76898cd6db4b53 in term 1.
I20260806 22:40:59.851146 25736 raft_consensus.cc:2454] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d75c199400b54951ac76898cd6db4b53 in term 1.
I20260806 22:40:59.851429 26032 leader_election.cc:302] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d75c199400b54951ac76898cd6db4b53, dec256cdd30b4020851faee43b550f8b; no voters:
I20260806 22:40:59.851766 26230 raft_consensus.cc:2781] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:40:59.852319 26230 raft_consensus.cc:686] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [term 1 LEADER]: Becoming Leader. State: Replica: d75c199400b54951ac76898cd6db4b53, State: Running, Role: LEADER
I20260806 22:40:59.852751 26230 consensus_queue.cc:237] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } }
I20260806 22:40:59.855269 26279 tablet_replica.cc:439] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: TxnStatusTablet state changed. Reason: New leader d75c199400b54951ac76898cd6db4b53. Latest consensus state: current_term: 1 leader_uuid: "d75c199400b54951ac76898cd6db4b53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } } }
I20260806 22:40:59.855670 26279 tablet_replica.cc:442] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: This TxnStatusTablet replica's current role is: LEADER
I20260806 22:40:59.856878 26281 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 22:40:59.856828 25504 catalog_manager.cc:5495] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 reported cstate change: term changed from 0 to 1, leader changed from <none> to d75c199400b54951ac76898cd6db4b53 (127.23.100.131). New cstate: current_term: 1 leader_uuid: "d75c199400b54951ac76898cd6db4b53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } health_report { overall_health: HEALTHY } } }
I20260806 22:40:59.857203 26281 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 22:40:59.892453 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.884306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.23.100.131:43785)
0806 22:40:59.884611 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.884619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.884709 (+ 90us) server_negotiation.cc:407] Connection header received
0806 22:40:59.885069 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.885075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.885184 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.885366 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:40:59.886005 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.887310 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.889292 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.890012 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.890144 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.890777 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:40:59.891585 (+ 808us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:40:59.891734 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.891906 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.892003 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260806 22:40:59.892413 26269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.884188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:43785 (local address 127.0.0.1:49246)
0806 22:40:59.884585 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.884607 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.884781 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.885607 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.885612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.885626 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 22:40:59.885831 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.885843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.887482 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.887489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.889117 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.889133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.890211 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.890218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.890449 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.890547 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 22:40:59.891731 (+ 1184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 22:40:59.891758 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.891853 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.892017 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260806 22:40:59.898443 25938 raft_consensus.cc:1261] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b [term 1 FOLLOWER]: Refusing update from remote peer d75c199400b54951ac76898cd6db4b53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:40:59.898590 25736 raft_consensus.cc:1261] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d [term 1 FOLLOWER]: Refusing update from remote peer d75c199400b54951ac76898cd6db4b53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:40:59.899480 26230 consensus_queue.cc:1035] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:40:59.899894 26279 consensus_queue.cc:1035] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:40:59.911842 26231 tablet_replica.cc:439] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d: TxnStatusTablet state changed. Reason: New leader d75c199400b54951ac76898cd6db4b53. Latest consensus state: current_term: 1 leader_uuid: "d75c199400b54951ac76898cd6db4b53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } } }
I20260806 22:40:59.912151 26231 tablet_replica.cc:442] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:40:59.914824 26245 tablet_replica.cc:439] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b: TxnStatusTablet state changed. Reason: New leader d75c199400b54951ac76898cd6db4b53. Latest consensus state: current_term: 1 leader_uuid: "d75c199400b54951ac76898cd6db4b53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } } }
I20260806 22:40:59.915186 26245 tablet_replica.cc:442] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:40:59.918450 26279 tablet_replica.cc:439] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d75c199400b54951ac76898cd6db4b53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } } }
I20260806 22:40:59.918740 26279 tablet_replica.cc:442] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: This TxnStatusTablet replica's current role is: LEADER
I20260806 22:40:59.924008 26230 tablet_replica.cc:439] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d75c199400b54951ac76898cd6db4b53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } } }
I20260806 22:40:59.924321 26230 tablet_replica.cc:442] T b01bd411326d4bd89630229c243bd4f4 P d75c199400b54951ac76898cd6db4b53: This TxnStatusTablet replica's current role is: LEADER
I20260806 22:40:59.921329 26245 tablet_replica.cc:439] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d75c199400b54951ac76898cd6db4b53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } } }
I20260806 22:40:59.925846 26245 tablet_replica.cc:442] T b01bd411326d4bd89630229c243bd4f4 P dec256cdd30b4020851faee43b550f8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:40:59.927022 26231 tablet_replica.cc:439] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d75c199400b54951ac76898cd6db4b53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 } } peers { permanent_uuid: "dec256cdd30b4020851faee43b550f8b" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44063 } } peers { permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 } } }
I20260806 22:40:59.927263 26231 tablet_replica.cc:442] T b01bd411326d4bd89630229c243bd4f4 P 4f4bedcef96f4a1099d4071d049df07d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:40:59.962984 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.955403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:36627 (local address 127.23.100.131:43785)
0806 22:40:59.955608 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 22:40:59.955614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:40:59.955780 (+ 166us) server_negotiation.cc:407] Connection header received
0806 22:40:59.956162 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:40:59.956168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:40:59.956260 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:40:59.956381 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:40:59.957067 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.958270 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.960592 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.961299 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.961452 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.962102 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:40:59.962274 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:40:59.962382 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 22:40:59.962564 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 22:40:59.962656 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 22:40:59.963136 26295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:40:59.954941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:43785 (local address 127.23.100.130:36627)
0806 22:40:59.955651 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0806 22:40:59.955672 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:40:59.955974 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:40:59.956421 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:40:59.956427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:40:59.956441 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 22:40:59.956876 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.956889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.958424 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.958431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.960411 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:40:59.960434 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:40:59.961548 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:40:59.961557 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:40:59.961790 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:40:59.961879 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 22:40:59.962400 (+ 521us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 22:40:59.962413 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 22:40:59.962510 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 22:40:59.962673 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":678,"threads_started":1}
I20260806 22:40:59.975004 25736 raft_consensus.cc:1261] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d [term 1 FOLLOWER]: Refusing update from remote peer dec256cdd30b4020851faee43b550f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:40:59.975006 26124 raft_consensus.cc:1261] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53 [term 1 FOLLOWER]: Refusing update from remote peer dec256cdd30b4020851faee43b550f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:40:59.975621 26245 consensus_queue.cc:1035] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d75c199400b54951ac76898cd6db4b53" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:40:59.975996 26232 consensus_queue.cc:1035] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f4bedcef96f4a1099d4071d049df07d" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:40:59.990973 26240 mvcc.cc:205] Tried to move back new op lower bound from 7315685621651738624 to 7315685620501471232. Current Snapshot: MvccSnapshot[applied={T|T < 7315685621651738624}]
I20260806 22:40:59.996831 26241 mvcc.cc:205] Tried to move back new op lower bound from 7315685621651738624 to 7315685620501471232. Current Snapshot: MvccSnapshot[applied={T|T < 7315685621651738624}]
I20260806 22:41:00.032989 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:00.025585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:44611 (local address 127.23.100.130:44063)
0806 22:41:00.025845 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:00.025855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:00.026575 (+ 720us) server_negotiation.cc:407] Connection header received
0806 22:41:00.026913 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:00.026919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:00.027011 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:00.027079 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:00.028053 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.029091 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.030737 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.031400 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.031500 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:00.032204 (+ 704us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:00.032355 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:00.032431 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:00.032681 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 22:41:00.032767 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 22:41:00.033813 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:00.025468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:44063 (local address 127.23.100.131:44611)
0806 22:41:00.026470 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:00.026492 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:00.026732 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:00.027294 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:00.027299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:00.027327 (+ 28us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 22:41:00.027819 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:00.027835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.029237 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.029240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:00.030546 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:00.030564 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.031708 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.031715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:00.031897 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:00.031975 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 22:41:00.032526 (+ 551us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 22:41:00.032540 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 22:41:00.033268 (+ 728us) client_negotiation.cc:240] Negotiation successful
0806 22:41:00.033409 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":613,"threads_started":1}
I20260806 22:41:00.084568 25503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:00.084865 25503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:00.088721 25503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b75f26fbde6c4aa19baabcc700172a3a: Sending DeleteTablet for 3 replicas of tablet c0998f4a56be4577a08775852ab02a89
I20260806 22:41:00.089676 25702 tablet_service.cc:1514] Processing DeleteTablet for tablet c0998f4a56be4577a08775852ab02a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:00 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:58596
I20260806 22:41:00.089993 26253 tablet_replica.cc:330] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d: stopping tablet replica
I20260806 22:41:00.090080 25899 tablet_service.cc:1514] Processing DeleteTablet for tablet c0998f4a56be4577a08775852ab02a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:00 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:46856
I20260806 22:41:00.090198 26253 raft_consensus.cc:2229] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:00.090364 26252 tablet_replica.cc:330] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b: stopping tablet replica
I20260806 22:41:00.090431 26253 raft_consensus.cc:2258] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:00.090576 26252 raft_consensus.cc:2229] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:00.090981 26252 raft_consensus.cc:2258] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:00.091130 26096 tablet_service.cc:1514] Processing DeleteTablet for tablet c0998f4a56be4577a08775852ab02a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:00 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:49240
I20260806 22:41:00.091482 26257 tablet_replica.cc:330] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53: stopping tablet replica
I20260806 22:41:00.091692 26257 raft_consensus.cc:2229] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:00.091929 26257 raft_consensus.cc:2258] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:00.092662 26252 ts_tablet_manager.cc:1905] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:00.093364 26253 ts_tablet_manager.cc:1905] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:00.100899 26252 ts_tablet_manager.cc:1918] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:00.101117 26252 log.cc:1196] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/c0998f4a56be4577a08775852ab02a89
I20260806 22:41:00.101886 26252 ts_tablet_manager.cc:1939] T c0998f4a56be4577a08775852ab02a89 P dec256cdd30b4020851faee43b550f8b: Deleting consensus metadata
I20260806 22:41:00.102929 26257 ts_tablet_manager.cc:1905] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:00.103544 25483 catalog_manager.cc:4841] TS dec256cdd30b4020851faee43b550f8b (127.23.100.130:44063): tablet c0998f4a56be4577a08775852ab02a89 (table test-table [id=57e8956403454b028c7554ea74575bc3]) successfully deleted
I20260806 22:41:00.107344 26253 ts_tablet_manager.cc:1918] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:00.108269 26253 log.cc:1196] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/c0998f4a56be4577a08775852ab02a89
I20260806 22:41:00.109261 26253 ts_tablet_manager.cc:1939] T c0998f4a56be4577a08775852ab02a89 P 4f4bedcef96f4a1099d4071d049df07d: Deleting consensus metadata
I20260806 22:41:00.110544 25481 catalog_manager.cc:4841] TS 4f4bedcef96f4a1099d4071d049df07d (127.23.100.129:43659): tablet c0998f4a56be4577a08775852ab02a89 (table test-table [id=57e8956403454b028c7554ea74575bc3]) successfully deleted
I20260806 22:41:00.111148 26257 ts_tablet_manager.cc:1918] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:00.111373 26257 log.cc:1196] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/c0998f4a56be4577a08775852ab02a89
I20260806 22:41:00.111930 26257 ts_tablet_manager.cc:1939] T c0998f4a56be4577a08775852ab02a89 P d75c199400b54951ac76898cd6db4b53: Deleting consensus metadata
I20260806 22:41:00.113169 25482 catalog_manager.cc:4841] TS d75c199400b54951ac76898cd6db4b53 (127.23.100.131:43785): tablet c0998f4a56be4577a08775852ab02a89 (table test-table [id=57e8956403454b028c7554ea74575bc3]) successfully deleted
Aug 06 22:41:00 dist-test-slave-zr1t krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:41:00.133211 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:00.107942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.23.100.129:43659)
0806 22:41:00.108137 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:00.108143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:00.114104 (+ 5961us) server_negotiation.cc:407] Connection header received
0806 22:41:00.114285 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:00.114291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:00.114396 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:00.114510 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:00.116394 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.117595 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.119448 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.120428 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.120559 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:00.125976 (+ 5417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:00.125997 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:00.126015 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:00.126059 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:00.129943 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:00.130848 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:00.130858 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:00.130862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:00.131049 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:00.131484 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:00.131489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:00.131492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:00.131937 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:00.132245 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:00.132557 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 22:41:00.132827 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 22:41:00.134239 25795 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58602
I20260806 22:41:00.155202 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:00.137444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49708 (local address 127.23.100.190:42951)
0806 22:41:00.137714 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:00.137721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:00.138506 (+ 785us) server_negotiation.cc:407] Connection header received
0806 22:41:00.138674 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:00.138681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:00.138780 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:00.138890 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:00.140712 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.141883 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.143807 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.144449 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.144584 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:00.147812 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:00.147821 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:00.147826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:00.147872 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:00.151817 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:00.152838 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:00.152844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:00.152848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:00.153017 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:00.153476 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:00.153482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:00.153485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:00.153901 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:00.154206 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:00.154484 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 22:41:00.154800 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
W20260806 22:41:00.156203 25503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49708
I20260806 22:41:00.158208 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25606
I20260806 22:41:00.177842 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:00.160746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.23.100.129:43659)
0806 22:41:00.160938 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:00.160945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:00.161606 (+ 661us) server_negotiation.cc:407] Connection header received
0806 22:41:00.161874 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:00.161881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:00.161966 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:00.162075 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:00.163574 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.164781 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.166604 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.167281 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.167414 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:00.170566 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:00.170575 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:00.170582 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:00.170631 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:00.174413 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:00.175335 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:00.175342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:00.175345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:00.175519 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:00.175989 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:00.175995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:00.175998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:00.176354 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:00.176621 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:00.176992 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 22:41:00.177228 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 22:41:00.178511 25795 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58618
W20260806 22:41:00.179215 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25606: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:41:00.179411 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 25606
I20260806 22:41:00.211921 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25805
Aug 06 22:41:00 dist-test-slave-zr1t krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:41:00.228287 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:00.212768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46868 (local address 127.23.100.130:44063)
0806 22:41:00.212950 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:00.212958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:00.212999 (+ 41us) server_negotiation.cc:407] Connection header received
0806 22:41:00.213230 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:00.213238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:00.213348 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:00.213466 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:00.214693 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.215464 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.216752 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.217161 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.217312 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:00.221941 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:00.221957 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:00.221973 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:00.222017 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:00.225315 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:00.226118 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:00.226126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:00.226128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:00.226278 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:00.226699 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:00.226703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:00.226706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:00.227070 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:00.227460 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:00.227678 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 22:41:00.227924 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 22:41:00.228958 25989 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46868
W20260806 22:41:00.229647 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25805: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:41:00.229833 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 25805
I20260806 22:41:00.273834 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 26004
Aug 06 22:41:00 dist-test-slave-zr1t krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:41:00.297091 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:00.274702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.23.100.131:43785)
0806 22:41:00.274914 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:00.274920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:00.275082 (+ 162us) server_negotiation.cc:407] Connection header received
0806 22:41:00.275293 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:00.275306 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:00.275401 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:00.275546 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:00.277297 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.278563 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.281651 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:00.282441 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:00.282581 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:00.288615 (+ 6034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:00.288634 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:00.288653 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:00.288716 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:00.293319 (+ 4603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:00.294228 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:00.294241 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:00.294245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:00.294444 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:00.294922 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:00.294928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:00.294934 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:00.295528 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:00.295874 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:00.296160 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 22:41:00.296477 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 22:41:00.297972 26187 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49254
W20260806 22:41:00.298928 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 26004: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:41:00.299109 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 26004
I20260806 22:41:00.328303 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25461
I20260806 22:41:00.329593 25565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:00.452594 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 25461
2026-08-06T22:41:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4839 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:41:00 dist-test-slave-zr1t krb5kdc[26314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:41:00 dist-test-slave-zr1t krb5kdc[26314](info): set up 2 sockets
Aug 06 22:41:00 dist-test-slave-zr1t krb5kdc[26314](info): commencing operation
krb5kdc: starting...
W20260806 22:41:01.821638 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.316s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:41:01 dist-test-slave-zr1t krb5kdc[26314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:41:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:41:02.061944 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:37829
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:37829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:02.405645 26335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:02.405961 26335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:02.406121 26335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:02.413568 26335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:02.413727 26335 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 22:41:02.413796 26335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:02.413889 26335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:02.413967 26335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:02.414049 26335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:02.424297 26335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:37829
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:37829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:02.425704 26335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:02.427153 26335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:02.439637 26342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:02.439687 26345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:02.439728 26343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:02.441090 26335 server_base.cc:1034] running on GCE node
I20260806 22:41:02.442134 26335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:02.443629 26335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:02.444865 26335 hybrid_clock.cc:648] HybridClock initialized: now 1786056062444804 us; error 37 us; skew 500 ppm
I20260806 22:41:02.445173 26335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:02 dist-test-slave-zr1t krb5kdc[26314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056062, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:02.448581 26335 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:41:02.450130 26335 webserver.cc:460] Webserver started at http://127.23.100.190:33559/ using document root <none> and password file <none>
I20260806 22:41:02.450826 26335 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:02.450930 26335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:02.451213 26335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:02.454345 26335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "a734716f094247ea848f5280b565127b"
format_stamp: "Formatted at 2026-08-06 22:41:02 on dist-test-slave-zr1t"
server_key: "7b832f8673b8e7fd09c235cb153b35e6"
server_key_iv: "045b567955f27dc8ffc1d15b2b70b5c0"
server_key_version: "encryptionkey@0"
I20260806 22:41:02.455204 26335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "a734716f094247ea848f5280b565127b"
format_stamp: "Formatted at 2026-08-06 22:41:02 on dist-test-slave-zr1t"
server_key: "7b832f8673b8e7fd09c235cb153b35e6"
server_key_iv: "045b567955f27dc8ffc1d15b2b70b5c0"
server_key_version: "encryptionkey@0"
I20260806 22:41:02.460758 26335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:41:02.464485 26354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:02.466131 26335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:02.466284 26335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "a734716f094247ea848f5280b565127b"
format_stamp: "Formatted at 2026-08-06 22:41:02 on dist-test-slave-zr1t"
server_key: "7b832f8673b8e7fd09c235cb153b35e6"
server_key_iv: "045b567955f27dc8ffc1d15b2b70b5c0"
server_key_version: "encryptionkey@0"
I20260806 22:41:02.466482 26335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:02.506398 26335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:02.511103 26335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:02.511353 26335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:02.534888 26335 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:37829
I20260806 22:41:02.534914 26434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:37829 every 8 connection(s)
I20260806 22:41:02.536370 26335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:02.536897 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 26335
I20260806 22:41:02.537138 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:02.538048 23954 external_mini_cluster.cc:1428] Setting key 51a905ac5992cdd723e81fe13f111fcc
I20260806 22:41:02.541244 26436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 22:41:02 dist-test-slave-zr1t krb5kdc[26314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:02.554379 26436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b: Bootstrap starting.
I20260806 22:41:02.558763 26436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:02.560075 26436 log.cc:824] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:02.563670 26436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b: No bootstrap required, opened a new log
I20260806 22:41:02.563961 26440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:02.540236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50684 (local address 127.23.100.190:37829)
0806 22:41:02.541061 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:02.541077 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:02.541126 (+ 49us) server_negotiation.cc:407] Connection header received
0806 22:41:02.543548 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:02.543576 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:02.544056 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:02.544532 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:02.546026 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:02.548145 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:02.549483 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:02.550530 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:02.550725 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:02.555071 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:02.555082 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:02.555101 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:02.555154 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:02.559280 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:02.560072 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:02.560081 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:02.560091 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:02.560297 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:02.560752 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:02.560760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:02.560766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:02.561377 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:02.561754 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:02.562173 (+ 419us) server_negotiation.cc:299] Negotiation successful
0806 22:41:02.562633 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":338,"threads_started":1}
I20260806 22:41:02.572252 26436 raft_consensus.cc:348] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a734716f094247ea848f5280b565127b" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37829 } }
I20260806 22:41:02.572667 26436 raft_consensus.cc:374] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:02.572777 26436 raft_consensus.cc:729] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a734716f094247ea848f5280b565127b, State: Initialized, Role: FOLLOWER
I20260806 22:41:02.573472 26436 consensus_queue.cc:260] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a734716f094247ea848f5280b565127b" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37829 } }
I20260806 22:41:02.573750 26436 raft_consensus.cc:388] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:02.573864 26436 raft_consensus.cc:482] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:02.573993 26436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:02.576151 26436 raft_consensus.cc:504] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a734716f094247ea848f5280b565127b" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37829 } }
I20260806 22:41:02.576647 26436 leader_election.cc:302] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a734716f094247ea848f5280b565127b; no voters:
I20260806 22:41:02.577241 26436 leader_election.cc:288] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:02.577421 26442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:02.578743 26442 raft_consensus.cc:686] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [term 1 LEADER]: Becoming Leader. State: Replica: a734716f094247ea848f5280b565127b, State: Running, Role: LEADER
I20260806 22:41:02.579345 26442 consensus_queue.cc:237] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a734716f094247ea848f5280b565127b" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37829 } }
I20260806 22:41:02.579566 26436 sys_catalog.cc:564] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:02.582583 26444 sys_catalog.cc:455] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a734716f094247ea848f5280b565127b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a734716f094247ea848f5280b565127b" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37829 } } }
I20260806 22:41:02.582959 26444 sys_catalog.cc:458] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:02.583510 26445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a734716f094247ea848f5280b565127b. Latest consensus state: current_term: 1 leader_uuid: "a734716f094247ea848f5280b565127b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a734716f094247ea848f5280b565127b" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37829 } } }
I20260806 22:41:02.583853 26445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:02.586627 26459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:02.591862 26459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:02.600399 26459 catalog_manager.cc:1347] Generated new cluster ID: b43319c27bf949b690207d440ea0a717
I20260806 22:41:02.600540 26459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:02.617172 26459 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:02.618872 26459 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:02.627878 26459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b: Generated new TSK 0
I20260806 22:41:02.629033 26459 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:41:02.683277 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37829
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:03.031333 26473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:03.031651 26473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:03.031760 26473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:03.039628 26473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:03.039754 26473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:03.039841 26473 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 22:41:03.039894 26473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 22:41:03.039988 26473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:41:03.051589 26473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.190:37829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:03.052911 26473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:03.054492 26473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:03.068289 26483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:03.068333 26478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:03.068393 26481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:03.069582 26473 server_base.cc:1034] running on GCE node
I20260806 22:41:03.070266 26473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:03.071244 26473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:03.072481 26473 hybrid_clock.cc:648] HybridClock initialized: now 1786056063072433 us; error 43 us; skew 500 ppm
I20260806 22:41:03.072803 26473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:03 dist-test-slave-zr1t krb5kdc[26314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056063, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:03.076243 26473 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:41:03.077829 26473 webserver.cc:460] Webserver started at http://127.23.100.129:33177/ using document root <none> and password file <none>
I20260806 22:41:03.078486 26473 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:03.078589 26473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:03.078853 26473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:03.081972 26473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "96be82b05cf14b12bc5a7cd3eb8d5367"
format_stamp: "Formatted at 2026-08-06 22:41:03 on dist-test-slave-zr1t"
server_key: "ebe80d18534c2deabe332dd4bde3a28a"
server_key_iv: "76b9983265245a64453441ee4942f840"
server_key_version: "encryptionkey@0"
I20260806 22:41:03.082718 26473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "96be82b05cf14b12bc5a7cd3eb8d5367"
format_stamp: "Formatted at 2026-08-06 22:41:03 on dist-test-slave-zr1t"
server_key: "ebe80d18534c2deabe332dd4bde3a28a"
server_key_iv: "76b9983265245a64453441ee4942f840"
server_key_version: "encryptionkey@0"
I20260806 22:41:03.087937 26473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 22:41:03.091379 26491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:03.092697 26473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 22:41:03.092864 26473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "96be82b05cf14b12bc5a7cd3eb8d5367"
format_stamp: "Formatted at 2026-08-06 22:41:03 on dist-test-slave-zr1t"
server_key: "ebe80d18534c2deabe332dd4bde3a28a"
server_key_iv: "76b9983265245a64453441ee4942f840"
server_key_version: "encryptionkey@0"
I20260806 22:41:03.093063 26473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:03.115204 26473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:03.119894 26473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:03.120158 26473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:03.123064 26473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:03.123211 26473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:03.123348 26473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:03.123440 26473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 22:41:03 dist-test-slave-zr1t krb5kdc[26314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056063, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:03.148075 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:03.123523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:36909 (local address 127.23.100.190:37829)
0806 22:41:03.124428 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:03.124432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:03.125194 (+ 762us) server_negotiation.cc:407] Connection header received
0806 22:41:03.127679 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:03.127686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:03.127770 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:03.127890 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:03.130241 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.131366 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.134823 (+ 3457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.135395 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.135577 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:03.140908 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:03.140917 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:03.140924 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:03.140989 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:03.144116 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:03.145305 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:03.145314 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:03.145321 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:03.145513 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:03.146186 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:03.146192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:03.146196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:03.146494 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:03.146658 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:03.147433 (+ 775us) server_negotiation.cc:299] Negotiation successful
0806 22:41:03.147686 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":695,"threads_started":1}
I20260806 22:41:03.149315 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:03.124211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37829 (local address 127.23.100.129:36909)
0806 22:41:03.125036 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:03.125080 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:03.127310 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:03.128142 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:03.128150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:03.128927 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:03.130009 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:03.130052 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.131601 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.131611 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:03.134610 (+ 2999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:03.134633 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.135727 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.135741 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:03.136166 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:03.137199 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:03.137243 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:03.140611 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:03.144288 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:03.144301 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:03.144327 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:03.145126 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:03.145700 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:03.145705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:03.145709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:03.146047 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:03.146629 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:03.146639 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:03.147038 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 22:41:03.147314 (+ 276us) client_negotiation.cc:240] Negotiation successful
0806 22:41:03.147895 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":378,"threads_started":1}
I20260806 22:41:03.171749 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:03.156478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:45111 (local address 127.23.100.190:37829)
0806 22:41:03.156729 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:03.156738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:03.157866 (+ 1128us) server_negotiation.cc:407] Connection header received
0806 22:41:03.158088 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:03.158093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:03.158159 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:03.158266 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:03.159297 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.160033 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.161418 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.161955 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.162115 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:03.165096 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:03.165101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:03.165104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:03.165138 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:03.168622 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:03.169369 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:03.169376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:03.169379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:03.169610 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:03.170094 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:03.170100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:03.170104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:03.170408 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:03.170572 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:03.171112 (+ 540us) server_negotiation.cc:299] Negotiation successful
0806 22:41:03.171354 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 22:41:03.171852 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:03.156961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37829 (local address 127.23.100.129:45111)
0806 22:41:03.157748 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:03.157768 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:03.157940 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:03.158297 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:03.158301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:03.158840 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:03.159140 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:03.159150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.160199 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.160205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:03.161261 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:03.161275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.162215 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.162226 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:03.162518 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:03.163307 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:03.163342 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:03.164837 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:03.168775 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:03.168780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:03.168785 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:03.169238 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:03.169763 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:03.169768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:03.169772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:03.169973 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:03.170548 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:03.170553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:03.170820 (+ 267us) client_negotiation.cc:769] Sending connection context
0806 22:41:03.170996 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 22:41:03.171372 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":456,"threads_started":1}
I20260806 22:41:03.184139 26473 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:37673
I20260806 22:41:03.184190 26661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:37673 every 8 connection(s)
I20260806 22:41:03.185683 26473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:41:03.187248 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 26473
I20260806 22:41:03.187561 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:41:03.188390 26662 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37829
I20260806 22:41:03.188549 23954 external_mini_cluster.cc:1428] Setting key c1c22732796607c0941907fe97c988a0
I20260806 22:41:03.188679 26662 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:03.189615 26662 heartbeater.cc:511] Master 127.23.100.190:37829 requested a full tablet report, sending...
I20260806 22:41:03.192229 26376 ts_manager.cc:194] Registered new tserver with Master: 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673)
I20260806 22:41:03.194514 26376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:36909
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:41:03.243417 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37829
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:03.588737 26668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:03.589035 26668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:03.589226 26668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:03.596807 26668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:03.596951 26668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:03.597110 26668 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 22:41:03.597185 26668 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 22:41:03.597301 26668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:41:03.607623 26668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.190:37829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:03.609027 26668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:03.610515 26668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:03.624199 26674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:03.624158 26677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:03.624320 26673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:03.625420 26668 server_base.cc:1034] running on GCE node
I20260806 22:41:03.626184 26668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:03.627245 26668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:03.628479 26668 hybrid_clock.cc:648] HybridClock initialized: now 1786056063628416 us; error 34 us; skew 500 ppm
I20260806 22:41:03.628844 26668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:03 dist-test-slave-zr1t krb5kdc[26314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056063, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:03.632179 26668 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:41:03.633790 26668 webserver.cc:460] Webserver started at http://127.23.100.130:41753/ using document root <none> and password file <none>
I20260806 22:41:03.634485 26668 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:03.634624 26668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:03.634974 26668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:03.638049 26668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "9cddf5c22aad441eb25ba3e9457a8053"
format_stamp: "Formatted at 2026-08-06 22:41:03 on dist-test-slave-zr1t"
server_key: "60c5293539fb512902785c9cb9f25c80"
server_key_iv: "e8c76d0b9069bd35d5b921fa8b2fb5f7"
server_key_version: "encryptionkey@0"
I20260806 22:41:03.638839 26668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "9cddf5c22aad441eb25ba3e9457a8053"
format_stamp: "Formatted at 2026-08-06 22:41:03 on dist-test-slave-zr1t"
server_key: "60c5293539fb512902785c9cb9f25c80"
server_key_iv: "e8c76d0b9069bd35d5b921fa8b2fb5f7"
server_key_version: "encryptionkey@0"
I20260806 22:41:03.644146 26668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:41:03.647835 26689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:03.649209 26668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:41:03.649398 26668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "9cddf5c22aad441eb25ba3e9457a8053"
format_stamp: "Formatted at 2026-08-06 22:41:03 on dist-test-slave-zr1t"
server_key: "60c5293539fb512902785c9cb9f25c80"
server_key_iv: "e8c76d0b9069bd35d5b921fa8b2fb5f7"
server_key_version: "encryptionkey@0"
I20260806 22:41:03.649691 26668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:03.687647 26668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:03.691913 26668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:03.692237 26668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:03.695113 26668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:03.695268 26668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:03.695420 26668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:03.695539 26668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 22:41:03 dist-test-slave-zr1t krb5kdc[26314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056063, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:03.723699 26702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:03.696180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37829 (local address 127.23.100.130:50611)
0806 22:41:03.697210 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:03.697256 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:03.700092 (+ 2836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:03.701065 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:03.701079 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:03.702120 (+ 1041us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:03.703243 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:03.703273 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.705111 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.705116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:03.707243 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:03.707261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.708584 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.708591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:03.708779 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:03.709645 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:03.709700 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:03.713976 (+ 4276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:03.718759 (+ 4783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:03.718775 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:03.718797 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:03.719502 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:03.720092 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:03.720105 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:03.720112 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:03.720521 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:03.721169 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:03.721178 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:03.721676 (+ 498us) client_negotiation.cc:769] Sending connection context
0806 22:41:03.721996 (+ 320us) client_negotiation.cc:240] Negotiation successful
0806 22:41:03.722589 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":432,"threads_started":1}
I20260806 22:41:03.724967 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:03.695478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:50611 (local address 127.23.100.190:37829)
0806 22:41:03.696037 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:03.696044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:03.697417 (+ 1373us) server_negotiation.cc:407] Connection header received
0806 22:41:03.700497 (+ 3080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:03.700506 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:03.700609 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:03.700747 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:03.703538 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.704932 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.707421 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.708276 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.708447 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:03.714343 (+ 5896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:03.714349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:03.714353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:03.714391 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:03.718567 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:03.719719 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:03.719726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:03.719730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:03.719934 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:03.720686 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:03.720694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:03.720700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:03.721026 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:03.721206 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:03.724513 (+ 3307us) server_negotiation.cc:299] Negotiation successful
0806 22:41:03.724707 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":259,"threads_started":1}
I20260806 22:41:03.745885 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:03.731770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:59009 (local address 127.23.100.190:37829)
0806 22:41:03.732010 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:03.732019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:03.732525 (+ 506us) server_negotiation.cc:407] Connection header received
0806 22:41:03.732680 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:03.732689 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:03.732796 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:03.732933 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:03.733974 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.734702 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.736033 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.736441 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.736593 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:03.738833 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:03.738837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:03.738841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:03.738871 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:03.742271 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:03.743207 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:03.743212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:03.743214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:03.743327 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:03.743887 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:03.743896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:03.743902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:03.744466 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:03.744646 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:03.745256 (+ 610us) server_negotiation.cc:299] Negotiation successful
0806 22:41:03.745491 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 22:41:03.746132 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:03.731754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37829 (local address 127.23.100.130:59009)
0806 22:41:03.732406 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:03.732426 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:03.732554 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:03.732960 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:03.732963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:03.733391 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:03.733800 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:03.733814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.734849 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.734855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:03.735869 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:03.735881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:03.736637 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:03.736643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:03.736806 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:03.737609 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:03.737628 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:03.738630 (+ 1002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:03.742412 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:03.742419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:03.742423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:03.743085 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:03.743473 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:03.743484 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:03.743487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:03.743713 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:03.744620 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:03.744625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:03.744913 (+ 288us) client_negotiation.cc:769] Sending connection context
0806 22:41:03.745163 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 22:41:03.745621 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":355,"threads_started":1}
I20260806 22:41:03.752453 26668 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:41031
I20260806 22:41:03.752550 26851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:41031 every 8 connection(s)
I20260806 22:41:03.754029 26668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:41:03.756099 26852 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37829
I20260806 22:41:03.756343 26852 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:03.756997 26852 heartbeater.cc:511] Master 127.23.100.190:37829 requested a full tablet report, sending...
I20260806 22:41:03.758385 26376 ts_manager.cc:194] Registered new tserver with Master: 9cddf5c22aad441eb25ba3e9457a8053 (127.23.100.130:41031)
I20260806 22:41:03.759049 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 26668
I20260806 22:41:03.759229 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:41:03.759565 26376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:50611
I20260806 22:41:03.759784 23954 external_mini_cluster.cc:1428] Setting key 4aef031f13d17b03285276b693d876aa
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:41:03.810062 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37829
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:04.164528 26859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:04.164883 26859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:04.165066 26859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:04.172746 26859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:04.172914 26859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:04.173058 26859 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 22:41:04.173146 26859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 22:41:04.173280 26859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:41:04.183734 26859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.190:37829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:04.185173 26859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:04.186780 26859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 22:41:04.198225 26662 heartbeater.cc:503] Master 127.23.100.190:37829 was elected leader, sending a full tablet report...
W20260806 22:41:04.200526 26866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:04.200480 26868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:04.201459 26865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:04.203178 26859 server_base.cc:1034] running on GCE node
I20260806 22:41:04.204136 26859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:04.205634 26859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:04.206908 26859 hybrid_clock.cc:648] HybridClock initialized: now 1786056064206807 us; error 81 us; skew 500 ppm
I20260806 22:41:04.207350 26859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:04 dist-test-slave-zr1t krb5kdc[26314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056064, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:04.215554 26859 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:41:04.217639 26859 webserver.cc:460] Webserver started at http://127.23.100.131:40983/ using document root <none> and password file <none>
I20260806 22:41:04.218550 26859 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:04.218739 26859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:04.219128 26859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:04.222733 26859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "07a1549e55234dd3a8c54b1bd9d25a64"
format_stamp: "Formatted at 2026-08-06 22:41:04 on dist-test-slave-zr1t"
server_key: "fa9f1e595574747ca4c8dc5a8a90565a"
server_key_iv: "eb8b4911707acae11ce333fd2bdc7a3e"
server_key_version: "encryptionkey@0"
I20260806 22:41:04.223671 26859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "07a1549e55234dd3a8c54b1bd9d25a64"
format_stamp: "Formatted at 2026-08-06 22:41:04 on dist-test-slave-zr1t"
server_key: "fa9f1e595574747ca4c8dc5a8a90565a"
server_key_iv: "eb8b4911707acae11ce333fd2bdc7a3e"
server_key_version: "encryptionkey@0"
I20260806 22:41:04.230086 26859 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260806 22:41:04.234242 26881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:04.235704 26859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:04.235915 26859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "07a1549e55234dd3a8c54b1bd9d25a64"
format_stamp: "Formatted at 2026-08-06 22:41:04 on dist-test-slave-zr1t"
server_key: "fa9f1e595574747ca4c8dc5a8a90565a"
server_key_iv: "eb8b4911707acae11ce333fd2bdc7a3e"
server_key_version: "encryptionkey@0"
I20260806 22:41:04.236153 26859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:04.263139 26859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:04.268358 26859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:04.268729 26859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:04.272102 26859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:04.272248 26859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:04.272409 26859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:04.272534 26859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 22:41:04 dist-test-slave-zr1t krb5kdc[26314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056064, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:04.301095 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.272671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:47625 (local address 127.23.100.190:37829)
0806 22:41:04.273218 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:04.273227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:04.274655 (+ 1428us) server_negotiation.cc:407] Connection header received
0806 22:41:04.276778 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:04.276784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:04.276885 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:04.277017 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:04.279432 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.280973 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.283974 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.284610 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.284759 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.291004 (+ 6245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:04.291013 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:04.291021 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:04.291080 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:04.295866 (+ 4786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:04.297342 (+ 1476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:04.297348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:04.297352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:04.297582 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:04.298315 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:04.298322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:04.298325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:04.298662 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:04.298855 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:04.300192 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0806 22:41:04.300520 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":285,"threads_started":1}
I20260806 22:41:04.303272 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.273605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37829 (local address 127.23.100.131:47625)
0806 22:41:04.274479 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:04.274527 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:04.276483 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:04.277295 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:04.277304 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:04.278172 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:04.279096 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.279148 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.281213 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.281221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.283701 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.283721 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.284884 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.284892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.285138 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.286180 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:04.286226 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:04.290635 (+ 4409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:04.296103 (+ 5468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:04.296118 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:04.296144 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:04.297088 (+ 944us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:04.297755 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:04.297763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:04.297772 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:04.298135 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:04.298824 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:04.298834 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:04.299537 (+ 703us) client_negotiation.cc:769] Sending connection context
0806 22:41:04.300016 (+ 479us) client_negotiation.cc:240] Negotiation successful
0806 22:41:04.300833 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":382,"threads_started":1}
I20260806 22:41:04.321228 26859 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:32993
I20260806 22:41:04.321346 27060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:32993 every 8 connection(s)
I20260806 22:41:04.322851 26859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:41:04.324158 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 26859
I20260806 22:41:04.324405 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:41:04.325062 23954 external_mini_cluster.cc:1428] Setting key d0b534737f5e5e568ee2f670a0ba7c70
I20260806 22:41:04.325564 27062 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37829
I20260806 22:41:04.325961 27062 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:04.327000 27062 heartbeater.cc:511] Master 127.23.100.190:37829 requested a full tablet report, sending...
I20260806 22:41:04.328837 26376 ts_manager.cc:194] Registered new tserver with Master: 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993)
I20260806 22:41:04.330174 26376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:47625
I20260806 22:41:04.334517 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.317182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37829 (local address 127.23.100.131:49811)
0806 22:41:04.318160 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:04.318179 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:04.318414 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:04.319025 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:04.319031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:04.319567 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:04.320083 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.320104 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.321761 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.321767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.322846 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.322862 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.325706 (+ 2844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.325718 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.325977 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.326775 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:04.326794 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:04.327981 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:04.331868 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:04.331874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:04.331878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:04.332362 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:04.332872 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:04.332876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:04.332878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:04.333010 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:04.333612 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:04.333623 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:04.333783 (+ 160us) client_negotiation.cc:769] Sending connection context
0806 22:41:04.333922 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 22:41:04.334180 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":577,"threads_started":1}
I20260806 22:41:04.334520 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.317006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:49811 (local address 127.23.100.190:37829)
0806 22:41:04.317196 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:04.317201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:04.318293 (+ 1092us) server_negotiation.cc:407] Connection header received
0806 22:41:04.318562 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:04.318570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:04.318673 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:04.318786 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:04.320301 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.321603 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.323007 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.323519 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.323662 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.328221 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:04.328228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:04.328233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:04.328286 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:04.331712 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:04.332594 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:04.332598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:04.332601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:04.332748 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:04.333118 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:04.333122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:04.333124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:04.333446 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:04.333652 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:04.334034 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 22:41:04.334230 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 22:41:04.341181 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 22:41:04 dist-test-slave-zr1t krb5kdc[26314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 22:41:04 dist-test-slave-zr1t krb5kdc[26314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:04.401266 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.385197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.23.100.190:37829)
0806 22:41:04.385507 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:04.385536 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:04.385898 (+ 362us) server_negotiation.cc:407] Connection header received
0806 22:41:04.386136 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:04.386141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:04.386224 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:04.386341 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:04.387984 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.388737 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.390135 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.390809 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.390995 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.395486 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:04.395492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:04.395496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:04.395540 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:04.398651 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:04.399394 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:04.399398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:04.399401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:04.399561 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:04.399920 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:04.399923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:04.399926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:04.400219 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:04.400438 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:04.400774 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 22:41:04.400982 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260806 22:41:04.406118 26376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:04.413198 26376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:04.445216 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.426010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:41031 (local address 127.0.0.1:35868)
0806 22:41:04.427189 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:04.427240 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:04.427471 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:04.434853 (+ 7382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:04.434861 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:04.434912 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:04.435494 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.435516 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.438109 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.438114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.442659 (+ 4545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.442672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.444064 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.444071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.444334 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.444403 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 22:41:04.444552 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 22:41:04.444773 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":523,"threads_started":1}
I20260806 22:41:04.446090 27084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.426267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.23.100.130:41031)
0806 22:41:04.434170 (+ 7903us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:04.434179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:04.434219 (+ 40us) server_negotiation.cc:407] Connection header received
0806 22:41:04.434313 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:04.434321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:04.434548 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:04.434825 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:04.435796 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.437812 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.442833 (+ 5021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.443773 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.444029 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.444257 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:04.444626 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 22:41:04.444819 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7731,"thread_start_us":7624,"threads_started":1}
I20260806 22:41:04.447283 27086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.426659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:32993 (local address 127.0.0.1:49538)
0806 22:41:04.436681 (+ 10022us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:04.436707 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:04.436856 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:04.439224 (+ 2368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:04.439228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:04.439257 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:04.439793 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.439807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.442007 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.442012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.444190 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.444203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.446389 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.446399 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.446596 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.446635 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 22:41:04.446745 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 22:41:04.446908 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9850,"spinlock_wait_cycles":18176,"thread_start_us":12436,"threads_started":1}
I20260806 22:41:04.447357 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.427343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.23.100.131:32993)
0806 22:41:04.438042 (+ 10699us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:04.438074 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:04.438125 (+ 51us) server_negotiation.cc:407] Connection header received
0806 22:41:04.438278 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:04.438290 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:04.438631 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:04.438861 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:04.440009 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.441808 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.444416 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.445982 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.446181 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.446362 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:04.446825 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 22:41:04.447013 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9984,"thread_start_us":9815,"threads_started":1}
I20260806 22:41:04.448818 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.427380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:37673 (local address 127.0.0.1:60532)
0806 22:41:04.439323 (+ 11943us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:04.439343 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:04.439519 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:04.440215 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:04.440219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:04.440244 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:04.440703 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.440715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.443875 (+ 3160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.443882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.446075 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.446092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.447983 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.447989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.448172 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.448202 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 22:41:04.448327 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 22:41:04.448462 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11783,"spinlock_wait_cycles":7424,"thread_start_us":10520,"threads_started":1}
I20260806 22:41:04.448848 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.428057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.23.100.129:37673)
0806 22:41:04.437989 (+ 9932us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:04.438002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:04.439651 (+ 1649us) server_negotiation.cc:407] Connection header received
0806 22:41:04.439781 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:04.439788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:04.440029 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:04.440236 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:04.440894 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.443707 (+ 2813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.446280 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.447786 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.447967 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.448127 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:04.448380 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 22:41:04.448553 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9727,"thread_start_us":9578,"threads_started":1}
I20260806 22:41:04.450702 26763 tablet_service.cc:1467] Processing CreateTablet for tablet c242fff4c16b48528ca4f1d30f56bb12 (DEFAULT_TABLE table=test-table [id=2939616751b44cb594ed5a91d41f516b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:04.451308 26961 tablet_service.cc:1467] Processing CreateTablet for tablet c242fff4c16b48528ca4f1d30f56bb12 (DEFAULT_TABLE table=test-table [id=2939616751b44cb594ed5a91d41f516b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:04.452258 26763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c242fff4c16b48528ca4f1d30f56bb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:04.452702 26961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c242fff4c16b48528ca4f1d30f56bb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:04.452613 26565 tablet_service.cc:1467] Processing CreateTablet for tablet c242fff4c16b48528ca4f1d30f56bb12 (DEFAULT_TABLE table=test-table [id=2939616751b44cb594ed5a91d41f516b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:04.454058 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c242fff4c16b48528ca4f1d30f56bb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:04.467272 27091 tablet_bootstrap.cc:492] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: Bootstrap starting.
I20260806 22:41:04.470484 27093 tablet_bootstrap.cc:492] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: Bootstrap starting.
I20260806 22:41:04.471084 27091 tablet_bootstrap.cc:654] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:04.472088 27092 tablet_bootstrap.cc:492] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: Bootstrap starting.
I20260806 22:41:04.472513 27091 log.cc:824] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:04.474431 27093 tablet_bootstrap.cc:654] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:04.475826 27092 tablet_bootstrap.cc:654] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:04.475800 27093 log.cc:824] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:04.477250 27092 log.cc:824] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:04.480266 27091 tablet_bootstrap.cc:492] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: No bootstrap required, opened a new log
I20260806 22:41:04.480396 27093 tablet_bootstrap.cc:492] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: No bootstrap required, opened a new log
I20260806 22:41:04.480556 27092 tablet_bootstrap.cc:492] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: No bootstrap required, opened a new log
I20260806 22:41:04.480623 27091 ts_tablet_manager.cc:1397] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260806 22:41:04.480698 27093 ts_tablet_manager.cc:1397] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260806 22:41:04.480809 27092 ts_tablet_manager.cc:1397] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 22:41:04.488781 27091 raft_consensus.cc:348] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.489131 27091 raft_consensus.cc:374] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:04.489198 27091 raft_consensus.cc:729] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96be82b05cf14b12bc5a7cd3eb8d5367, State: Initialized, Role: FOLLOWER
I20260806 22:41:04.489915 27091 consensus_queue.cc:260] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.491837 27091 ts_tablet_manager.cc:1428] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260806 22:41:04.491731 27093 raft_consensus.cc:348] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.491724 27092 raft_consensus.cc:348] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.492179 27092 raft_consensus.cc:374] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:04.492202 27093 raft_consensus.cc:374] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:04.492312 27092 raft_consensus.cc:729] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cddf5c22aad441eb25ba3e9457a8053, State: Initialized, Role: FOLLOWER
I20260806 22:41:04.492302 27093 raft_consensus.cc:729] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a1549e55234dd3a8c54b1bd9d25a64, State: Initialized, Role: FOLLOWER
I20260806 22:41:04.493021 27093 consensus_queue.cc:260] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.493023 27092 consensus_queue.cc:260] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.494597 27062 heartbeater.cc:503] Master 127.23.100.190:37829 was elected leader, sending a full tablet report...
I20260806 22:41:04.495064 27093 ts_tablet_manager.cc:1428] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260806 22:41:04.496716 26852 heartbeater.cc:503] Master 127.23.100.190:37829 was elected leader, sending a full tablet report...
I20260806 22:41:04.497300 27092 ts_tablet_manager.cc:1428] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260806 22:41:04.506295 26856 tablet.cc:2369] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:41:04.575119 27063 tablet.cc:2369] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:41:04.688445 26663 tablet.cc:2369] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:41:04.832702 27103 raft_consensus.cc:482] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:04.833101 27103 raft_consensus.cc:504] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.835067 27103 leader_election.cc:288] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673), 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993)
I20260806 22:41:04.843267 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.835572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:38071 (local address 127.23.100.131:32993)
0806 22:41:04.835792 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:04.835802 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:04.836594 (+ 792us) server_negotiation.cc:407] Connection header received
0806 22:41:04.836811 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:04.836818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:04.836907 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:04.837008 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:04.837725 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.838893 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.841247 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.842193 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.842356 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.842410 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:04.842857 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 22:41:04.843000 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 22:41:04.843297 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.835367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:32993 (local address 127.23.100.130:38071)
0806 22:41:04.836463 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:04.836479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:04.836643 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:04.837042 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:04.837045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:04.837067 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:04.837561 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.837576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.839215 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.839224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.841076 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.841091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.842448 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.842456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.842644 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.842684 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 22:41:04.842799 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 22:41:04.842912 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":887,"threads_started":1}
I20260806 22:41:04.843523 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.835759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:45275 (local address 127.23.100.129:37673)
0806 22:41:04.836050 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:04.836056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:04.836645 (+ 589us) server_negotiation.cc:407] Connection header received
0806 22:41:04.837348 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:04.837354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:04.837437 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:04.837575 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:04.838177 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.839145 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.841480 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.842491 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.842658 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.842726 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:04.843097 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 22:41:04.843231 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 22:41:04.843906 27104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.835514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:37673 (local address 127.23.100.130:45275)
0806 22:41:04.836505 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:04.836520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:04.837179 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:04.837600 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:04.837603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:04.837623 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:04.838009 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.838021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.839487 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.839491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.841253 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.841268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.842709 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.842713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.842907 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.842940 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 22:41:04.843034 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 22:41:04.843173 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"thread_start_us":492,"threads_started":1}
I20260806 22:41:04.844524 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c242fff4c16b48528ca4f1d30f56bb12" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" is_pre_election: true
I20260806 22:41:04.845129 26987 raft_consensus.cc:2454] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 0.
I20260806 22:41:04.845060 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c242fff4c16b48528ca4f1d30f56bb12" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" is_pre_election: true
I20260806 22:41:04.845665 26599 raft_consensus.cc:2454] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 0.
I20260806 22:41:04.845985 26695 leader_election.cc:302] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07a1549e55234dd3a8c54b1bd9d25a64, 9cddf5c22aad441eb25ba3e9457a8053; no voters:
I20260806 22:41:04.846555 27103 raft_consensus.cc:2781] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:04.846684 27103 raft_consensus.cc:482] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:04.846760 27103 raft_consensus.cc:3037] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:04.848927 27103 raft_consensus.cc:504] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.849583 27103 leader_election.cc:288] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 election: Requested vote from peers 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673), 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993)
I20260806 22:41:04.850067 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c242fff4c16b48528ca4f1d30f56bb12" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367"
I20260806 22:41:04.850159 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c242fff4c16b48528ca4f1d30f56bb12" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07a1549e55234dd3a8c54b1bd9d25a64"
I20260806 22:41:04.850274 26599 raft_consensus.cc:3037] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:04.850442 26987 raft_consensus.cc:3037] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:04.852725 26599 raft_consensus.cc:2454] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 1.
I20260806 22:41:04.853027 26987 raft_consensus.cc:2454] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 1.
I20260806 22:41:04.853284 26691 leader_election.cc:302] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96be82b05cf14b12bc5a7cd3eb8d5367, 9cddf5c22aad441eb25ba3e9457a8053; no voters:
I20260806 22:41:04.853662 27103 raft_consensus.cc:2781] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:04.854184 27103 raft_consensus.cc:686] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 LEADER]: Becoming Leader. State: Replica: 9cddf5c22aad441eb25ba3e9457a8053, State: Running, Role: LEADER
I20260806 22:41:04.854998 27103 consensus_queue.cc:237] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.859668 26375 catalog_manager.cc:5495] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cddf5c22aad441eb25ba3e9457a8053 (127.23.100.130). New cstate: current_term: 1 leader_uuid: "9cddf5c22aad441eb25ba3e9457a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } health_report { overall_health: UNKNOWN } } }
Aug 06 22:41:04 dist-test-slave-zr1t krb5kdc[26314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056062, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:04.938910 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.924172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37829 (local address 127.0.0.1:50704)
0806 22:41:04.925056 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:04.925071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:04.925239 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:04.925605 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:04.925612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:04.926441 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:04.926782 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.926793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.927998 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.928005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.928956 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:04.928968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.929776 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.929782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:04.929972 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.930643 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:04.930660 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:04.932949 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:04.936351 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:04.936365 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:04.936372 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:04.936867 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:04.937270 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:04.937274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:04.937276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:04.937429 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:04.938050 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:04.938058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:04.938211 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 22:41:04.938356 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 22:41:04.938607 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":563,"threads_started":1}
I20260806 22:41:04.939154 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:04.924114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.23.100.190:37829)
0806 22:41:04.924905 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:04.924916 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:04.925157 (+ 241us) server_negotiation.cc:407] Connection header received
0806 22:41:04.925362 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:04.925367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:04.925428 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:04.925568 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:04.926956 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.927877 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.929113 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:04.929589 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:04.929730 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:04.933207 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:04.933212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:04.933216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:04.933249 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:04.936180 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:04.937008 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:04.937012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:04.937015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:04.937155 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:04.937625 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:04.937629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:04.937631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:04.937933 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:04.938052 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:04.938467 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 22:41:04.938781 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":569,"thread_start_us":383,"threads_started":1}
I20260806 22:41:04.944715 26375 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:50704:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 22:41:04.945853 26375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:04.952734 26565 tablet_service.cc:1467] Processing CreateTablet for tablet 36b3ccf94c5a4a888bbb133b97a6acb2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6100df5ccf3742779c89dc73a3381294]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 22:41:04.952734 26763 tablet_service.cc:1467] Processing CreateTablet for tablet 36b3ccf94c5a4a888bbb133b97a6acb2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6100df5ccf3742779c89dc73a3381294]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 22:41:04.952991 26961 tablet_service.cc:1467] Processing CreateTablet for tablet 36b3ccf94c5a4a888bbb133b97a6acb2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6100df5ccf3742779c89dc73a3381294]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 22:41:04.953773 26763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b3ccf94c5a4a888bbb133b97a6acb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:04.953773 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b3ccf94c5a4a888bbb133b97a6acb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:04.953867 26961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36b3ccf94c5a4a888bbb133b97a6acb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:04.959852 27093 tablet_bootstrap.cc:492] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64: Bootstrap starting.
I20260806 22:41:04.960999 27092 tablet_bootstrap.cc:492] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: Bootstrap starting.
I20260806 22:41:04.961412 27091 tablet_bootstrap.cc:492] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367: Bootstrap starting.
I20260806 22:41:04.961479 27093 tablet_bootstrap.cc:654] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:04.962667 27091 tablet_bootstrap.cc:654] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:04.962857 27092 tablet_bootstrap.cc:654] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:04.965102 27091 tablet_bootstrap.cc:492] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367: No bootstrap required, opened a new log
I20260806 22:41:04.965137 27093 tablet_bootstrap.cc:492] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64: No bootstrap required, opened a new log
I20260806 22:41:04.965250 27091 ts_tablet_manager.cc:1397] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 22:41:04.965283 27093 ts_tablet_manager.cc:1397] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260806 22:41:04.965770 27092 tablet_bootstrap.cc:492] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: No bootstrap required, opened a new log
I20260806 22:41:04.965927 27092 ts_tablet_manager.cc:1397] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 22:41:04.966128 27091 raft_consensus.cc:348] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.966125 27093 raft_consensus.cc:348] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.966349 27091 raft_consensus.cc:374] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:04.966370 27093 raft_consensus.cc:374] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:04.966439 27091 raft_consensus.cc:729] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96be82b05cf14b12bc5a7cd3eb8d5367, State: Initialized, Role: FOLLOWER
I20260806 22:41:04.966476 27093 raft_consensus.cc:729] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a1549e55234dd3a8c54b1bd9d25a64, State: Initialized, Role: FOLLOWER
I20260806 22:41:04.966676 27091 consensus_queue.cc:260] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.966706 27093 consensus_queue.cc:260] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.967247 27091 ts_tablet_manager.cc:1428] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 22:41:04.967357 27093 ts_tablet_manager.cc:1428] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 22:41:04.967195 27102 tablet_replica.cc:439] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } } }
I20260806 22:41:04.967562 27102 tablet_replica.cc:442] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:04.966804 27092 raft_consensus.cc:348] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.968075 27092 raft_consensus.cc:374] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:04.968538 27092 raft_consensus.cc:729] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cddf5c22aad441eb25ba3e9457a8053, State: Initialized, Role: FOLLOWER
I20260806 22:41:04.968937 27092 consensus_queue.cc:260] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.968858 27100 tablet_replica.cc:439] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } } }
I20260806 22:41:04.969244 27100 tablet_replica.cc:442] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:04.969419 27103 tablet_replica.cc:439] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } } }
I20260806 22:41:04.969719 27103 tablet_replica.cc:442] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:04.970252 27092 ts_tablet_manager.cc:1428] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 22:41:04.990089 27103 raft_consensus.cc:482] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:04.990356 27103 raft_consensus.cc:504] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.991062 27103 leader_election.cc:288] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673), 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993)
I20260806 22:41:04.991554 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b3ccf94c5a4a888bbb133b97a6acb2" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" is_pre_election: true
I20260806 22:41:04.991622 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b3ccf94c5a4a888bbb133b97a6acb2" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" is_pre_election: true
I20260806 22:41:04.991842 26599 raft_consensus.cc:2454] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 0.
I20260806 22:41:04.991925 26987 raft_consensus.cc:2454] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 0.
I20260806 22:41:04.992261 26691 leader_election.cc:302] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96be82b05cf14b12bc5a7cd3eb8d5367, 9cddf5c22aad441eb25ba3e9457a8053; no voters:
I20260806 22:41:04.992590 27103 raft_consensus.cc:2781] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:04.992703 27103 raft_consensus.cc:482] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:04.992826 27103 raft_consensus.cc:3037] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:04.995082 27103 raft_consensus.cc:504] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:04.995659 27103 leader_election.cc:288] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 election: Requested vote from peers 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673), 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993)
I20260806 22:41:04.996155 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b3ccf94c5a4a888bbb133b97a6acb2" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07a1549e55234dd3a8c54b1bd9d25a64"
I20260806 22:41:04.996392 26987 raft_consensus.cc:3037] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:04.996464 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36b3ccf94c5a4a888bbb133b97a6acb2" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367"
I20260806 22:41:04.996744 26599 raft_consensus.cc:3037] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:04.998759 26987 raft_consensus.cc:2454] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 1.
I20260806 22:41:04.998962 26599 raft_consensus.cc:2454] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 1.
I20260806 22:41:04.999238 26695 leader_election.cc:302] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07a1549e55234dd3a8c54b1bd9d25a64, 9cddf5c22aad441eb25ba3e9457a8053; no voters:
I20260806 22:41:04.999548 27103 raft_consensus.cc:2781] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:04.999719 27103 raft_consensus.cc:686] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 LEADER]: Becoming Leader. State: Replica: 9cddf5c22aad441eb25ba3e9457a8053, State: Running, Role: LEADER
I20260806 22:41:05.000000 27103 consensus_queue.cc:237] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:05.001698 27106 tablet_replica.cc:439] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: TxnStatusTablet state changed. Reason: New leader 9cddf5c22aad441eb25ba3e9457a8053. Latest consensus state: current_term: 1 leader_uuid: "9cddf5c22aad441eb25ba3e9457a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } } }
I20260806 22:41:05.002014 27106 tablet_replica.cc:442] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: This TxnStatusTablet replica's current role is: LEADER
I20260806 22:41:05.002832 27124 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 22:41:05.003180 27124 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 22:41:05.003294 26376 catalog_manager.cc:5495] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cddf5c22aad441eb25ba3e9457a8053 (127.23.100.130). New cstate: current_term: 1 leader_uuid: "9cddf5c22aad441eb25ba3e9457a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } health_report { overall_health: UNKNOWN } } }
I20260806 22:41:05.035928 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:05.028640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.23.100.130:41031)
0806 22:41:05.029262 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:05.029266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:05.029288 (+ 22us) server_negotiation.cc:407] Connection header received
0806 22:41:05.029380 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:05.029383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:05.029463 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:05.029672 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:05.030188 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:05.031097 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:05.033111 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:05.033706 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:05.033820 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:05.034519 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:05.035175 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:05.035304 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:05.035477 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 22:41:05.035558 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":357,"threads_started":1}
I20260806 22:41:05.035928 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:05.028534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:41031 (local address 127.0.0.1:35884)
0806 22:41:05.028914 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:05.028937 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:05.029084 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:05.029726 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:05.029732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:05.029747 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 22:41:05.029967 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:05.029979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:05.031248 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:05.031254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:05.032934 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:05.032953 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:05.033879 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:05.033885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:05.034117 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:05.034205 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 22:41:05.035302 (+ 1097us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 22:41:05.035328 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 22:41:05.035426 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 22:41:05.035598 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260806 22:41:05.045648 26987 raft_consensus.cc:1261] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cddf5c22aad441eb25ba3e9457a8053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:05.046283 26599 raft_consensus.cc:1261] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Refusing update from remote peer 9cddf5c22aad441eb25ba3e9457a8053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:05.046988 27106 consensus_queue.cc:1035] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:05.047551 27103 consensus_queue.cc:1035] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:05.061164 27100 tablet_replica.cc:439] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367: TxnStatusTablet state changed. Reason: New leader 9cddf5c22aad441eb25ba3e9457a8053. Latest consensus state: current_term: 1 leader_uuid: "9cddf5c22aad441eb25ba3e9457a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } } }
I20260806 22:41:05.061511 27100 tablet_replica.cc:442] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:05.063643 27102 tablet_replica.cc:439] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64: TxnStatusTablet state changed. Reason: New leader 9cddf5c22aad441eb25ba3e9457a8053. Latest consensus state: current_term: 1 leader_uuid: "9cddf5c22aad441eb25ba3e9457a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } } }
I20260806 22:41:05.063956 27102 tablet_replica.cc:442] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:05.064338 27131 mvcc.cc:205] Tried to move back new op lower bound from 7315685642414067712 to 7315685642242236416. Current Snapshot: MvccSnapshot[applied={T|T < 7315685642414067712}]
I20260806 22:41:05.071069 27106 tablet_replica.cc:439] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cddf5c22aad441eb25ba3e9457a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } } }
I20260806 22:41:05.071550 27106 tablet_replica.cc:442] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: This TxnStatusTablet replica's current role is: LEADER
I20260806 22:41:05.075191 27106 tablet_replica.cc:439] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cddf5c22aad441eb25ba3e9457a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } } }
I20260806 22:41:05.075572 27106 tablet_replica.cc:442] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 9cddf5c22aad441eb25ba3e9457a8053: This TxnStatusTablet replica's current role is: LEADER
I20260806 22:41:05.079908 27102 tablet_replica.cc:439] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cddf5c22aad441eb25ba3e9457a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } } }
I20260806 22:41:05.080196 27102 tablet_replica.cc:442] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 07a1549e55234dd3a8c54b1bd9d25a64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:05.082096 27100 tablet_replica.cc:439] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cddf5c22aad441eb25ba3e9457a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } } }
I20260806 22:41:05.082365 27100 tablet_replica.cc:442] T 36b3ccf94c5a4a888bbb133b97a6acb2 P 96be82b05cf14b12bc5a7cd3eb8d5367: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:05.092682 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:05.086066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.23.100.130:41031)
0806 22:41:05.086249 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:05.086256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:05.086878 (+ 622us) server_negotiation.cc:407] Connection header received
0806 22:41:05.087029 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:05.087035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:05.087123 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:05.087225 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:05.087707 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:05.088906 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:05.090499 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:05.091203 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:05.091341 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:05.091865 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:05.092010 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:05.092101 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:05.092263 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 22:41:05.092350 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 22:41:05.111523 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:05.104457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:42509 (local address 127.23.100.130:41031)
0806 22:41:05.104689 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:05.104695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:05.105454 (+ 759us) server_negotiation.cc:407] Connection header received
0806 22:41:05.105680 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:05.105686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:05.105795 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:05.105930 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:05.106581 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:05.107767 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:05.109249 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:05.109979 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:05.110105 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:05.110623 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:05.110759 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:05.110852 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:05.111012 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 22:41:05.111136 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":61}
I20260806 22:41:05.111588 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:05.104509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:41031 (local address 127.23.100.130:42509)
0806 22:41:05.105342 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:05.105357 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:05.105495 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:05.106026 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:05.106029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:05.106040 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 22:41:05.106402 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:05.106411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:05.107902 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:05.107906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:05.109107 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:05.109120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:05.110184 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:05.110189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:05.110340 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:05.110402 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 22:41:05.110861 (+ 459us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 22:41:05.110894 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 22:41:05.110964 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 22:41:05.111101 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":3,"thread_start_us":467,"threads_started":1}
I20260806 22:41:05.122042 26599 raft_consensus.cc:1261] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Refusing update from remote peer 9cddf5c22aad441eb25ba3e9457a8053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:05.122149 26987 raft_consensus.cc:1261] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cddf5c22aad441eb25ba3e9457a8053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:05.122766 27103 consensus_queue.cc:1035] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:05.123099 27145 consensus_queue.cc:1035] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:05.190909 26376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:05.191233 26376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:05.194926 26376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b: Sending DeleteTablet for 3 replicas of tablet c242fff4c16b48528ca4f1d30f56bb12
I20260806 22:41:05.196282 26565 tablet_service.cc:1514] Processing DeleteTablet for tablet c242fff4c16b48528ca4f1d30f56bb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:05 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:60532
I20260806 22:41:05.197293 26961 tablet_service.cc:1514] Processing DeleteTablet for tablet c242fff4c16b48528ca4f1d30f56bb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:05 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:49538
I20260806 22:41:05.197381 27150 tablet_replica.cc:330] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: stopping tablet replica
I20260806 22:41:05.197918 27150 raft_consensus.cc:2229] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:05.198401 27150 raft_consensus.cc:2258] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:05.198607 26763 tablet_service.cc:1514] Processing DeleteTablet for tablet c242fff4c16b48528ca4f1d30f56bb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:05 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:35868
I20260806 22:41:05.200801 27150 ts_tablet_manager.cc:1905] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:05.204578 26376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:05.205354 26376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:05.215170 26960 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1eaf57703b45618e0290440a0dcc7f (DEFAULT_TABLE table=test-table [id=8da775fbe4254da9a6b3f18134a4bf49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:05.215765 26960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1eaf57703b45618e0290440a0dcc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:05.218086 27150 ts_tablet_manager.cc:1918] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:05.218331 27150 log.cc:1196] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/c242fff4c16b48528ca4f1d30f56bb12
I20260806 22:41:05.218397 26565 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1eaf57703b45618e0290440a0dcc7f (DEFAULT_TABLE table=test-table [id=8da775fbe4254da9a6b3f18134a4bf49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:05.218873 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1eaf57703b45618e0290440a0dcc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:05.218982 27150 ts_tablet_manager.cc:1939] T c242fff4c16b48528ca4f1d30f56bb12 P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting consensus metadata
I20260806 22:41:05.218835 26761 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1eaf57703b45618e0290440a0dcc7f (DEFAULT_TABLE table=test-table [id=8da775fbe4254da9a6b3f18134a4bf49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:05.219441 26761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1eaf57703b45618e0290440a0dcc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:05.221935 26355 catalog_manager.cc:4841] TS 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673): tablet c242fff4c16b48528ca4f1d30f56bb12 (table test-table [id=2939616751b44cb594ed5a91d41f516b]) successfully deleted
I20260806 22:41:05.225736 27093 tablet_bootstrap.cc:492] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64: Bootstrap starting.
I20260806 22:41:05.227506 27093 tablet_bootstrap.cc:654] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:05.225481 27091 tablet_bootstrap.cc:492] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367: Bootstrap starting.
I20260806 22:41:05.230252 27093 tablet_bootstrap.cc:492] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64: No bootstrap required, opened a new log
I20260806 22:41:05.230304 27091 tablet_bootstrap.cc:654] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:05.230437 27093 ts_tablet_manager.cc:1397] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 22:41:05.231421 27093 raft_consensus.cc:348] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:05.231839 27093 raft_consensus.cc:374] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:05.231990 27093 raft_consensus.cc:729] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a1549e55234dd3a8c54b1bd9d25a64, State: Initialized, Role: FOLLOWER
I20260806 22:41:05.232406 27093 consensus_queue.cc:260] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:05.234230 27091 tablet_bootstrap.cc:492] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367: No bootstrap required, opened a new log
I20260806 22:41:05.235087 27091 ts_tablet_manager.cc:1397] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260806 22:41:05.235297 27092 tablet_bootstrap.cc:492] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053: Bootstrap starting.
I20260806 22:41:05.235792 27091 raft_consensus.cc:348] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:05.236044 27091 raft_consensus.cc:374] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:05.236135 27091 raft_consensus.cc:729] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96be82b05cf14b12bc5a7cd3eb8d5367, State: Initialized, Role: FOLLOWER
I20260806 22:41:05.236382 27091 consensus_queue.cc:260] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:05.237040 27093 ts_tablet_manager.cc:1428] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 22:41:05.237085 27091 ts_tablet_manager.cc:1428] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 22:41:05.237556 27092 tablet_bootstrap.cc:654] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:05.238006 27151 tablet_replica.cc:330] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: stopping tablet replica
I20260806 22:41:05.238497 27151 raft_consensus.cc:2229] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:05.238907 27151 raft_consensus.cc:2258] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:05.239887 27152 tablet_replica.cc:330] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: stopping tablet replica
I20260806 22:41:05.240331 27152 raft_consensus.cc:2229] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:05.241068 27152 raft_consensus.cc:2258] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:05.243223 27152 ts_tablet_manager.cc:1905] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:05.247771 27151 ts_tablet_manager.cc:1905] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:05.251502 27152 ts_tablet_manager.cc:1918] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:05.251783 27152 log.cc:1196] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/c242fff4c16b48528ca4f1d30f56bb12
I20260806 22:41:05.252511 27152 ts_tablet_manager.cc:1939] T c242fff4c16b48528ca4f1d30f56bb12 P 9cddf5c22aad441eb25ba3e9457a8053: Deleting consensus metadata
I20260806 22:41:05.254001 27092 tablet_bootstrap.cc:492] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053: No bootstrap required, opened a new log
I20260806 22:41:05.254184 27092 ts_tablet_manager.cc:1397] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20260806 22:41:05.254448 26355 catalog_manager.cc:4841] TS 9cddf5c22aad441eb25ba3e9457a8053 (127.23.100.130:41031): tablet c242fff4c16b48528ca4f1d30f56bb12 (table test-table [id=2939616751b44cb594ed5a91d41f516b]) successfully deleted
I20260806 22:41:05.255043 27092 raft_consensus.cc:348] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:05.255306 27092 raft_consensus.cc:374] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:05.255395 27092 raft_consensus.cc:729] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cddf5c22aad441eb25ba3e9457a8053, State: Initialized, Role: FOLLOWER
I20260806 22:41:05.255685 27092 consensus_queue.cc:260] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:05.256340 27092 ts_tablet_manager.cc:1428] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 22:41:05.257907 27151 ts_tablet_manager.cc:1918] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:05.258113 27151 log.cc:1196] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/c242fff4c16b48528ca4f1d30f56bb12
I20260806 22:41:05.258708 27151 ts_tablet_manager.cc:1939] T c242fff4c16b48528ca4f1d30f56bb12 P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting consensus metadata
I20260806 22:41:05.260448 26359 catalog_manager.cc:4841] TS 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993): tablet c242fff4c16b48528ca4f1d30f56bb12 (table test-table [id=2939616751b44cb594ed5a91d41f516b]) successfully deleted
I20260806 22:41:05.419260 27103 raft_consensus.cc:482] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:05.419539 27103 raft_consensus.cc:504] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:05.420313 27103 leader_election.cc:288] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673), 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993)
I20260806 22:41:05.420984 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1eaf57703b45618e0290440a0dcc7f" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" is_pre_election: true
I20260806 22:41:05.421036 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1eaf57703b45618e0290440a0dcc7f" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" is_pre_election: true
I20260806 22:41:05.421262 26599 raft_consensus.cc:2454] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 0.
I20260806 22:41:05.421273 26987 raft_consensus.cc:2454] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 0.
I20260806 22:41:05.421833 26691 leader_election.cc:302] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96be82b05cf14b12bc5a7cd3eb8d5367, 9cddf5c22aad441eb25ba3e9457a8053; no voters:
I20260806 22:41:05.422175 27103 raft_consensus.cc:2781] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:05.422341 27103 raft_consensus.cc:482] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:05.422480 27103 raft_consensus.cc:3037] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:05.424799 27103 raft_consensus.cc:504] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:05.425402 27103 leader_election.cc:288] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 election: Requested vote from peers 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673), 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993)
I20260806 22:41:05.425968 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1eaf57703b45618e0290440a0dcc7f" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07a1549e55234dd3a8c54b1bd9d25a64"
I20260806 22:41:05.425989 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1eaf57703b45618e0290440a0dcc7f" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367"
I20260806 22:41:05.426214 26987 raft_consensus.cc:3037] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:05.426226 26599 raft_consensus.cc:3037] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:05.428741 26987 raft_consensus.cc:2454] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 1.
I20260806 22:41:05.429205 26695 leader_election.cc:302] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07a1549e55234dd3a8c54b1bd9d25a64, 9cddf5c22aad441eb25ba3e9457a8053; no voters:
I20260806 22:41:05.429330 26599 raft_consensus.cc:2454] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 1.
I20260806 22:41:05.429591 27103 raft_consensus.cc:2781] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:05.429813 27103 raft_consensus.cc:686] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 LEADER]: Becoming Leader. State: Replica: 9cddf5c22aad441eb25ba3e9457a8053, State: Running, Role: LEADER
I20260806 22:41:05.430095 27103 consensus_queue.cc:237] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:05.432940 26376 catalog_manager.cc:5495] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cddf5c22aad441eb25ba3e9457a8053 (127.23.100.130). New cstate: current_term: 1 leader_uuid: "9cddf5c22aad441eb25ba3e9457a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } health_report { overall_health: UNKNOWN } } }
I20260806 22:41:05.497123 26987 raft_consensus.cc:1261] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cddf5c22aad441eb25ba3e9457a8053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:05.497571 26590 raft_consensus.cc:1261] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Refusing update from remote peer 9cddf5c22aad441eb25ba3e9457a8053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:05.497797 27106 consensus_queue.cc:1035] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:05.498164 27145 consensus_queue.cc:1035] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:05.513473 27133 mvcc.cc:205] Tried to move back new op lower bound from 7315685644271312896 to 7315685644003930112. Current Snapshot: MvccSnapshot[applied={T|T < 7315685644271312896}]
W20260806 22:41:05.532241 27131 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 22:41:07.127316 26376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:07.127555 26376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:07.132030 26376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b: Sending DeleteTablet for 3 replicas of tablet 6f1eaf57703b45618e0290440a0dcc7f
I20260806 22:41:07.133004 26565 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1eaf57703b45618e0290440a0dcc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:07 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:60532
I20260806 22:41:07.133023 26763 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1eaf57703b45618e0290440a0dcc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:07 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:35868
I20260806 22:41:07.133069 26961 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f1eaf57703b45618e0290440a0dcc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:07 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:49538
I20260806 22:41:07.133857 27187 tablet_replica.cc:330] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053: stopping tablet replica
I20260806 22:41:07.134209 27187 raft_consensus.cc:2229] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:07.134809 27187 raft_consensus.cc:2258] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:07.135617 27188 tablet_replica.cc:330] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367: stopping tablet replica
I20260806 22:41:07.136677 27187 ts_tablet_manager.cc:1905] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:07.138271 27188 raft_consensus.cc:2229] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:07.138666 27188 raft_consensus.cc:2258] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:07.140414 27188 ts_tablet_manager.cc:1905] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:07.145968 27187 ts_tablet_manager.cc:1918] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 22:41:07.146230 27187 log.cc:1196] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/6f1eaf57703b45618e0290440a0dcc7f
I20260806 22:41:07.146937 27187 ts_tablet_manager.cc:1939] T 6f1eaf57703b45618e0290440a0dcc7f P 9cddf5c22aad441eb25ba3e9457a8053: Deleting consensus metadata
I20260806 22:41:07.148345 26355 catalog_manager.cc:4841] TS 9cddf5c22aad441eb25ba3e9457a8053 (127.23.100.130:41031): tablet 6f1eaf57703b45618e0290440a0dcc7f (table test-table [id=8da775fbe4254da9a6b3f18134a4bf49]) successfully deleted
I20260806 22:41:07.148406 27189 tablet_replica.cc:330] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64: stopping tablet replica
I20260806 22:41:07.148640 27189 raft_consensus.cc:2229] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:07.148898 27189 raft_consensus.cc:2258] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:07.149495 27188 ts_tablet_manager.cc:1918] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 22:41:07.149670 27188 log.cc:1196] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/6f1eaf57703b45618e0290440a0dcc7f
I20260806 22:41:07.150002 27189 ts_tablet_manager.cc:1905] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:07.150156 27188 ts_tablet_manager.cc:1939] T 6f1eaf57703b45618e0290440a0dcc7f P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting consensus metadata
I20260806 22:41:07.151437 26355 catalog_manager.cc:4841] TS 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673): tablet 6f1eaf57703b45618e0290440a0dcc7f (table test-table [id=8da775fbe4254da9a6b3f18134a4bf49]) successfully deleted
I20260806 22:41:07.155469 27189 ts_tablet_manager.cc:1918] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 22:41:07.155640 27189 log.cc:1196] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/6f1eaf57703b45618e0290440a0dcc7f
I20260806 22:41:07.156083 27189 ts_tablet_manager.cc:1939] T 6f1eaf57703b45618e0290440a0dcc7f P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting consensus metadata
I20260806 22:41:07.157174 26359 catalog_manager.cc:4841] TS 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993): tablet 6f1eaf57703b45618e0290440a0dcc7f (table test-table [id=8da775fbe4254da9a6b3f18134a4bf49]) successfully deleted
Aug 06 22:41:07 dist-test-slave-zr1t krb5kdc[26314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 22:41:07 dist-test-slave-zr1t krb5kdc[26314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:07.199913 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:07.183160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.23.100.190:37829)
0806 22:41:07.183779 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:07.183784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:07.183808 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:41:07.184124 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:07.184128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:07.184195 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:07.184299 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:07.186106 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.187020 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.188859 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.189255 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.189357 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:07.194042 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:07.194047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:07.194050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:07.194086 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:07.197021 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:07.197903 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:07.197915 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:07.197919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:07.198049 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:07.198445 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:07.198450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:07.198452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:07.198734 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:07.199031 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:07.199326 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 22:41:07.199559 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":336,"threads_started":1}
I20260806 22:41:07.203644 26376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:07.204406 26376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:07.210882 26565 tablet_service.cc:1467] Processing CreateTablet for tablet 2a58c8c7757c41d08f5ad1ecb0bb710e (DEFAULT_TABLE table=test-table [id=a49d0a1e6aab467fb9f5b16918788e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:07.211192 26763 tablet_service.cc:1467] Processing CreateTablet for tablet 2a58c8c7757c41d08f5ad1ecb0bb710e (DEFAULT_TABLE table=test-table [id=a49d0a1e6aab467fb9f5b16918788e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:07.211189 26961 tablet_service.cc:1467] Processing CreateTablet for tablet 2a58c8c7757c41d08f5ad1ecb0bb710e (DEFAULT_TABLE table=test-table [id=a49d0a1e6aab467fb9f5b16918788e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:07.211331 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a58c8c7757c41d08f5ad1ecb0bb710e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:07.211611 26763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a58c8c7757c41d08f5ad1ecb0bb710e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:07.211647 26961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a58c8c7757c41d08f5ad1ecb0bb710e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:07.218216 27202 tablet_bootstrap.cc:492] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64: Bootstrap starting.
I20260806 22:41:07.219710 27202 tablet_bootstrap.cc:654] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:07.222196 27203 tablet_bootstrap.cc:492] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053: Bootstrap starting.
I20260806 22:41:07.223059 27204 tablet_bootstrap.cc:492] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367: Bootstrap starting.
I20260806 22:41:07.224500 27204 tablet_bootstrap.cc:654] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:07.225033 27203 tablet_bootstrap.cc:654] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:07.227166 27202 tablet_bootstrap.cc:492] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64: No bootstrap required, opened a new log
I20260806 22:41:07.227372 27202 ts_tablet_manager.cc:1397] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260806 22:41:07.227535 27204 tablet_bootstrap.cc:492] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367: No bootstrap required, opened a new log
I20260806 22:41:07.227705 27204 ts_tablet_manager.cc:1397] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 22:41:07.228287 27203 tablet_bootstrap.cc:492] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053: No bootstrap required, opened a new log
I20260806 22:41:07.228215 27202 raft_consensus.cc:348] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.228448 27203 ts_tablet_manager.cc:1397] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 22:41:07.228463 27202 raft_consensus.cc:374] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:07.228564 27202 raft_consensus.cc:729] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a1549e55234dd3a8c54b1bd9d25a64, State: Initialized, Role: FOLLOWER
I20260806 22:41:07.228587 27204 raft_consensus.cc:348] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.228842 27204 raft_consensus.cc:374] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:07.228940 27204 raft_consensus.cc:729] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96be82b05cf14b12bc5a7cd3eb8d5367, State: Initialized, Role: FOLLOWER
I20260806 22:41:07.228848 27202 consensus_queue.cc:260] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.229189 27204 consensus_queue.cc:260] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.229436 27203 raft_consensus.cc:348] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.229822 27203 raft_consensus.cc:374] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:07.229851 27202 ts_tablet_manager.cc:1428] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 22:41:07.230005 27203 raft_consensus.cc:729] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cddf5c22aad441eb25ba3e9457a8053, State: Initialized, Role: FOLLOWER
I20260806 22:41:07.230335 27203 consensus_queue.cc:260] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.230862 27204 ts_tablet_manager.cc:1428] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:07.231680 27203 ts_tablet_manager.cc:1428] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:07.260964 27164 raft_consensus.cc:482] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:07.261227 27164 raft_consensus.cc:504] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.262012 27164 leader_election.cc:288] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673), 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993)
I20260806 22:41:07.262687 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a58c8c7757c41d08f5ad1ecb0bb710e" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" is_pre_election: true
I20260806 22:41:07.262706 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a58c8c7757c41d08f5ad1ecb0bb710e" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" is_pre_election: true
I20260806 22:41:07.262993 26599 raft_consensus.cc:2454] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 0.
I20260806 22:41:07.262979 26987 raft_consensus.cc:2454] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 0.
I20260806 22:41:07.263571 26695 leader_election.cc:302] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07a1549e55234dd3a8c54b1bd9d25a64, 9cddf5c22aad441eb25ba3e9457a8053; no voters:
I20260806 22:41:07.263909 27164 raft_consensus.cc:2781] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:07.264158 27164 raft_consensus.cc:482] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:07.264309 27164 raft_consensus.cc:3037] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:07.266681 27164 raft_consensus.cc:504] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.267293 27164 leader_election.cc:288] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 election: Requested vote from peers 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673), 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993)
I20260806 22:41:07.268005 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a58c8c7757c41d08f5ad1ecb0bb710e" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367"
I20260806 22:41:07.268276 26599 raft_consensus.cc:3037] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:07.269856 26987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a58c8c7757c41d08f5ad1ecb0bb710e" candidate_uuid: "9cddf5c22aad441eb25ba3e9457a8053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07a1549e55234dd3a8c54b1bd9d25a64"
I20260806 22:41:07.270153 26987 raft_consensus.cc:3037] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:07.271235 26599 raft_consensus.cc:2454] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 1.
I20260806 22:41:07.271823 26691 leader_election.cc:302] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96be82b05cf14b12bc5a7cd3eb8d5367, 9cddf5c22aad441eb25ba3e9457a8053; no voters:
I20260806 22:41:07.272188 27164 raft_consensus.cc:2781] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:07.272382 27164 raft_consensus.cc:686] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 LEADER]: Becoming Leader. State: Replica: 9cddf5c22aad441eb25ba3e9457a8053, State: Running, Role: LEADER
I20260806 22:41:07.272696 27164 consensus_queue.cc:237] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.273502 26987 raft_consensus.cc:2454] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cddf5c22aad441eb25ba3e9457a8053 in term 1.
I20260806 22:41:07.275585 26376 catalog_manager.cc:5495] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cddf5c22aad441eb25ba3e9457a8053 (127.23.100.130). New cstate: current_term: 1 leader_uuid: "9cddf5c22aad441eb25ba3e9457a8053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } health_report { overall_health: UNKNOWN } } }
I20260806 22:41:07.314863 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:07.309418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.23.100.130:41031)
0806 22:41:07.309945 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:07.309956 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:07.309976 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:41:07.310031 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:07.310034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:07.310097 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:07.310181 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:07.310533 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.311357 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.312807 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.313262 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.313365 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:07.313878 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:07.314001 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:07.314073 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:07.314282 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 22:41:07.314353 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":291,"threads_started":1}
I20260806 22:41:07.324487 26987 raft_consensus.cc:1261] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cddf5c22aad441eb25ba3e9457a8053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:07.324579 26599 raft_consensus.cc:1261] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Refusing update from remote peer 9cddf5c22aad441eb25ba3e9457a8053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:07.325052 27173 consensus_queue.cc:1035] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:07.325403 27164 consensus_queue.cc:1035] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:07.338402 27165 mvcc.cc:205] Tried to move back new op lower bound from 7315685651756511232 to 7315685651551113216. Current Snapshot: MvccSnapshot[applied={T|T < 7315685651756511232}]
I20260806 22:41:07.376896 26374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:07.377149 26374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:07.380466 26374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b: Sending DeleteTablet for 3 replicas of tablet 2a58c8c7757c41d08f5ad1ecb0bb710e
I20260806 22:41:07.381412 26565 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a58c8c7757c41d08f5ad1ecb0bb710e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:07 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:60532
I20260806 22:41:07.381465 26763 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a58c8c7757c41d08f5ad1ecb0bb710e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:07 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:35868
I20260806 22:41:07.381603 26961 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a58c8c7757c41d08f5ad1ecb0bb710e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:07 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:49538
I20260806 22:41:07.381775 27188 tablet_replica.cc:330] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367: stopping tablet replica
I20260806 22:41:07.381775 27187 tablet_replica.cc:330] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053: stopping tablet replica
I20260806 22:41:07.381901 27189 tablet_replica.cc:330] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64: stopping tablet replica
I20260806 22:41:07.381995 27188 raft_consensus.cc:2229] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:07.381995 27187 raft_consensus.cc:2229] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:07.382109 27189 raft_consensus.cc:2229] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:07.382202 27188 raft_consensus.cc:2258] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:07.382333 27189 raft_consensus.cc:2258] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:07.382372 27187 raft_consensus.cc:2258] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:07.383587 27188 ts_tablet_manager.cc:1905] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:07.383620 27189 ts_tablet_manager.cc:1905] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:07.385972 27187 ts_tablet_manager.cc:1905] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:07.385792 26374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:07.386713 26374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:07.390797 27188 ts_tablet_manager.cc:1918] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:07.391002 27188 log.cc:1196] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/2a58c8c7757c41d08f5ad1ecb0bb710e
I20260806 22:41:07.391530 27188 ts_tablet_manager.cc:1939] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting consensus metadata
I20260806 22:41:07.392984 27189 ts_tablet_manager.cc:1918] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:07.393175 27189 log.cc:1196] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/2a58c8c7757c41d08f5ad1ecb0bb710e
I20260806 22:41:07.393179 26355 catalog_manager.cc:4841] TS 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673): tablet 2a58c8c7757c41d08f5ad1ecb0bb710e (table test-table [id=a49d0a1e6aab467fb9f5b16918788e3a]) successfully deleted
I20260806 22:41:07.393695 27189 ts_tablet_manager.cc:1939] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting consensus metadata
I20260806 22:41:07.394145 27187 ts_tablet_manager.cc:1918] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:07.394340 27187 log.cc:1196] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/2a58c8c7757c41d08f5ad1ecb0bb710e
I20260806 22:41:07.394891 27187 ts_tablet_manager.cc:1939] T 2a58c8c7757c41d08f5ad1ecb0bb710e P 9cddf5c22aad441eb25ba3e9457a8053: Deleting consensus metadata
I20260806 22:41:07.394999 26359 catalog_manager.cc:4841] TS 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993): tablet 2a58c8c7757c41d08f5ad1ecb0bb710e (table test-table [id=a49d0a1e6aab467fb9f5b16918788e3a]) successfully deleted
I20260806 22:41:07.395211 26961 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec5bc6523954763a1848a6a84efbdf4 (DEFAULT_TABLE table=test-table [id=0cdf076a797d4961b55a46cb88991a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:07.395651 26961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec5bc6523954763a1848a6a84efbdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:07.395787 26763 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec5bc6523954763a1848a6a84efbdf4 (DEFAULT_TABLE table=test-table [id=0cdf076a797d4961b55a46cb88991a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:07.395810 26565 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec5bc6523954763a1848a6a84efbdf4 (DEFAULT_TABLE table=test-table [id=0cdf076a797d4961b55a46cb88991a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:07.396195 26763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec5bc6523954763a1848a6a84efbdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:07.396456 26565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec5bc6523954763a1848a6a84efbdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:07.396394 26355 catalog_manager.cc:4841] TS 9cddf5c22aad441eb25ba3e9457a8053 (127.23.100.130:41031): tablet 2a58c8c7757c41d08f5ad1ecb0bb710e (table test-table [id=a49d0a1e6aab467fb9f5b16918788e3a]) successfully deleted
I20260806 22:41:07.402524 27204 tablet_bootstrap.cc:492] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367: Bootstrap starting.
I20260806 22:41:07.402624 27202 tablet_bootstrap.cc:492] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64: Bootstrap starting.
I20260806 22:41:07.402669 27203 tablet_bootstrap.cc:492] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053: Bootstrap starting.
I20260806 22:41:07.404237 27202 tablet_bootstrap.cc:654] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:07.404500 27203 tablet_bootstrap.cc:654] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:07.404531 27204 tablet_bootstrap.cc:654] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:07.406634 27202 tablet_bootstrap.cc:492] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64: No bootstrap required, opened a new log
I20260806 22:41:07.406819 27202 ts_tablet_manager.cc:1397] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260806 22:41:07.407655 27202 raft_consensus.cc:348] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.407896 27202 raft_consensus.cc:374] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:07.407979 27202 raft_consensus.cc:729] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a1549e55234dd3a8c54b1bd9d25a64, State: Initialized, Role: FOLLOWER
I20260806 22:41:07.408258 27202 consensus_queue.cc:260] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.408798 27202 ts_tablet_manager.cc:1428] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260806 22:41:07.412786 27204 tablet_bootstrap.cc:492] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367: No bootstrap required, opened a new log
I20260806 22:41:07.413060 27204 ts_tablet_manager.cc:1397] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260806 22:41:07.414283 27204 raft_consensus.cc:348] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.414683 27204 raft_consensus.cc:374] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:07.414794 27204 raft_consensus.cc:729] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96be82b05cf14b12bc5a7cd3eb8d5367, State: Initialized, Role: FOLLOWER
I20260806 22:41:07.415151 27204 consensus_queue.cc:260] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.415933 27204 ts_tablet_manager.cc:1428] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:07.416877 27203 tablet_bootstrap.cc:492] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053: No bootstrap required, opened a new log
I20260806 22:41:07.417040 27203 ts_tablet_manager.cc:1397] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260806 22:41:07.417865 27203 raft_consensus.cc:348] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.418113 27203 raft_consensus.cc:374] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:07.418226 27203 raft_consensus.cc:729] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cddf5c22aad441eb25ba3e9457a8053, State: Initialized, Role: FOLLOWER
I20260806 22:41:07.418547 27203 consensus_queue.cc:260] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.419114 27203 ts_tablet_manager.cc:1428] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 22:41:07.443917 27213 raft_consensus.cc:482] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:07.444238 27213 raft_consensus.cc:504] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.445928 27213 leader_election.cc:288] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673), 9cddf5c22aad441eb25ba3e9457a8053 (127.23.100.130:41031)
I20260806 22:41:07.453958 27230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:07.446269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:37673 (local address 127.23.100.131:45929)
0806 22:41:07.446966 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:07.446990 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:07.447132 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:07.447611 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:07.447617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:07.447662 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:07.448049 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:07.448066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.449491 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.449498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:07.451333 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:07.451347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.453002 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.453009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:07.453217 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:07.453257 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 22:41:07.453368 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 22:41:07.453524 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":308,"threads_started":1}
I20260806 22:41:07.454228 27231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:07.446401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:45929 (local address 127.23.100.129:37673)
0806 22:41:07.447077 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:07.447085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:07.447119 (+ 34us) server_negotiation.cc:407] Connection header received
0806 22:41:07.447306 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:07.447313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:07.447399 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:07.448118 (+ 719us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:07.448276 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.449323 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.451521 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.452786 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.452945 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:07.453000 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:07.453432 (+ 432us) server_negotiation.cc:299] Negotiation successful
0806 22:41:07.453597 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":379,"threads_started":1}
I20260806 22:41:07.454696 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:07.447357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:59173 (local address 127.23.100.130:41031)
0806 22:41:07.447574 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:07.447581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:07.447986 (+ 405us) server_negotiation.cc:407] Connection header received
0806 22:41:07.448111 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:07.448120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:07.448214 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:07.448310 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:07.449011 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.450046 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.452264 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.453334 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.453610 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:07.453686 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:07.454202 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 22:41:07.454372 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 22:41:07.454782 27232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:07.447135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:41031 (local address 127.23.100.131:59173)
0806 22:41:07.447858 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:07.447876 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:07.447995 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:07.448362 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:07.448368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:07.448429 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:07.448797 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:07.448810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.450209 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.450217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:07.451983 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:07.451999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.453824 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.453833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:07.454027 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:07.454071 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 22:41:07.454171 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 22:41:07.454336 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":1,"thread_start_us":390,"threads_started":1}
I20260806 22:41:07.454795 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec5bc6523954763a1848a6a84efbdf4" candidate_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" is_pre_election: true
I20260806 22:41:07.455103 26599 raft_consensus.cc:2454] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07a1549e55234dd3a8c54b1bd9d25a64 in term 0.
I20260806 22:41:07.455729 26882 leader_election.cc:302] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07a1549e55234dd3a8c54b1bd9d25a64, 96be82b05cf14b12bc5a7cd3eb8d5367; no voters:
I20260806 22:41:07.455714 26790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec5bc6523954763a1848a6a84efbdf4" candidate_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cddf5c22aad441eb25ba3e9457a8053" is_pre_election: true
I20260806 22:41:07.456193 27213 raft_consensus.cc:2781] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:07.456176 26790 raft_consensus.cc:2454] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07a1549e55234dd3a8c54b1bd9d25a64 in term 0.
I20260806 22:41:07.456318 27213 raft_consensus.cc:482] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:07.456410 27213 raft_consensus.cc:3037] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:07.458752 27213 raft_consensus.cc:504] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.459388 27213 leader_election.cc:288] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [CANDIDATE]: Term 1 election: Requested vote from peers 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673), 9cddf5c22aad441eb25ba3e9457a8053 (127.23.100.130:41031)
I20260806 22:41:07.460088 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec5bc6523954763a1848a6a84efbdf4" candidate_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367"
I20260806 22:41:07.460311 26599 raft_consensus.cc:3037] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:07.460228 26790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec5bc6523954763a1848a6a84efbdf4" candidate_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cddf5c22aad441eb25ba3e9457a8053"
I20260806 22:41:07.460493 26790 raft_consensus.cc:3037] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:07.462888 26790 raft_consensus.cc:2454] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07a1549e55234dd3a8c54b1bd9d25a64 in term 1.
I20260806 22:41:07.462896 26599 raft_consensus.cc:2454] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07a1549e55234dd3a8c54b1bd9d25a64 in term 1.
I20260806 22:41:07.463387 26882 leader_election.cc:302] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07a1549e55234dd3a8c54b1bd9d25a64, 96be82b05cf14b12bc5a7cd3eb8d5367; no voters:
I20260806 22:41:07.463752 27213 raft_consensus.cc:2781] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:07.464268 27213 raft_consensus.cc:686] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 LEADER]: Becoming Leader. State: Replica: 07a1549e55234dd3a8c54b1bd9d25a64, State: Running, Role: LEADER
I20260806 22:41:07.464668 27213 consensus_queue.cc:237] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } }
I20260806 22:41:07.468230 26376 catalog_manager.cc:5495] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131). New cstate: current_term: 1 leader_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07a1549e55234dd3a8c54b1bd9d25a64" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 32993 } health_report { overall_health: HEALTHY } } }
I20260806 22:41:07.502955 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:07.496929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.23.100.131:32993)
0806 22:41:07.497632 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:07.497637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:07.497658 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:41:07.497740 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:07.497743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:07.497819 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:07.497979 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:07.498366 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.499155 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.500501 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.500904 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.501002 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:07.501599 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:07.502266 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:07.502404 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:07.502567 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 22:41:07.502647 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":362,"threads_started":1}
I20260806 22:41:07.519066 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:07.513618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:41031 (local address 127.23.100.131:34705)
0806 22:41:07.514321 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:07.514337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:07.514461 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:07.514801 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:07.514804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:07.514813 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 22:41:07.515111 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:07.515118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.516143 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.516146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:07.517212 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:07.517223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.518048 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.518052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:07.518179 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:07.518236 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 22:41:07.518596 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 22:41:07.518604 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 22:41:07.518662 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 22:41:07.518758 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":358,"threads_started":1}
I20260806 22:41:07.519083 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:07.513729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:34705 (local address 127.23.100.130:41031)
0806 22:41:07.513927 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:07.513932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:07.514415 (+ 483us) server_negotiation.cc:407] Connection header received
0806 22:41:07.514579 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:07.514583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:07.514689 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:07.514779 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:07.515254 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.516029 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.517342 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.517871 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.517979 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:07.518394 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:07.518509 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:07.518590 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:07.518703 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 22:41:07.518769 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 22:41:07.528000 26599 raft_consensus.cc:1261] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Refusing update from remote peer 07a1549e55234dd3a8c54b1bd9d25a64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:07.528039 26790 raft_consensus.cc:1261] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Refusing update from remote peer 07a1549e55234dd3a8c54b1bd9d25a64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:07.528740 27234 consensus_queue.cc:1035] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96be82b05cf14b12bc5a7cd3eb8d5367" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:07.529129 27213 consensus_queue.cc:1035] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cddf5c22aad441eb25ba3e9457a8053" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:07.564435 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:07.555223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:45811 (local address 127.23.100.131:32993)
0806 22:41:07.555484 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:07.555495 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:07.558268 (+ 2773us) server_negotiation.cc:407] Connection header received
0806 22:41:07.558614 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:07.558620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:07.558730 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:07.558826 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:07.559443 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.560239 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.562258 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.562895 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.563011 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:07.563556 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:07.563734 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:07.563845 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:07.564044 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 22:41:07.564138 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 22:41:07.564769 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:07.555161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:32993 (local address 127.23.100.130:45811)
0806 22:41:07.558149 (+ 2988us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:07.558179 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:07.558395 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:07.558863 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:07.558869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:07.558883 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 22:41:07.559239 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:07.559252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.560391 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.560397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:07.562082 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:07.562100 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:07.563092 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:07.563099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:07.563291 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:07.563350 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 22:41:07.563859 (+ 509us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 22:41:07.563871 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 22:41:07.564002 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 22:41:07.564157 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2807,"thread_start_us":2634,"threads_started":1}
W20260806 22:41:07.565946 27165 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 22:41:09.252238 26374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:09.252589 26374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:09.257234 26374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a734716f094247ea848f5280b565127b: Sending DeleteTablet for 3 replicas of tablet 0ec5bc6523954763a1848a6a84efbdf4
I20260806 22:41:09.258143 26565 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec5bc6523954763a1848a6a84efbdf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:09 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:60532
I20260806 22:41:09.258324 26763 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec5bc6523954763a1848a6a84efbdf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:09 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:35868
I20260806 22:41:09.258777 26961 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec5bc6523954763a1848a6a84efbdf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:09 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:49538
I20260806 22:41:09.259057 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 26473
I20260806 22:41:09.267578 27286 tablet_replica.cc:330] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367: stopping tablet replica
I20260806 22:41:09.267800 27288 tablet_replica.cc:330] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64: stopping tablet replica
I20260806 22:41:09.268002 27288 raft_consensus.cc:2229] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:09.268004 27286 raft_consensus.cc:2229] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:09.268347 27286 raft_consensus.cc:2258] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:09.268659 27288 raft_consensus.cc:2258] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:09.269620 27286 ts_tablet_manager.cc:1905] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:09.269769 27288 ts_tablet_manager.cc:1905] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:09.271379 27287 tablet_replica.cc:330] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053: stopping tablet replica
I20260806 22:41:09.271591 27287 raft_consensus.cc:2229] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:09.271822 27287 raft_consensus.cc:2258] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:09.272821 27287 ts_tablet_manager.cc:1905] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:09.277310 27288 ts_tablet_manager.cc:1918] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 22:41:09.277362 27286 ts_tablet_manager.cc:1918] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 22:41:09.277473 27288 log.cc:1196] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/0ec5bc6523954763a1848a6a84efbdf4
I20260806 22:41:09.277583 27286 log.cc:1196] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/0ec5bc6523954763a1848a6a84efbdf4
I20260806 22:41:09.277976 27288 ts_tablet_manager.cc:1939] T 0ec5bc6523954763a1848a6a84efbdf4 P 07a1549e55234dd3a8c54b1bd9d25a64: Deleting consensus metadata
I20260806 22:41:09.278132 27286 ts_tablet_manager.cc:1939] T 0ec5bc6523954763a1848a6a84efbdf4 P 96be82b05cf14b12bc5a7cd3eb8d5367: Deleting consensus metadata
I20260806 22:41:09.279194 26359 catalog_manager.cc:4841] TS 07a1549e55234dd3a8c54b1bd9d25a64 (127.23.100.131:32993): tablet 0ec5bc6523954763a1848a6a84efbdf4 (table test-table [id=0cdf076a797d4961b55a46cb88991a5d]) successfully deleted
I20260806 22:41:09.279479 26355 catalog_manager.cc:4841] TS 96be82b05cf14b12bc5a7cd3eb8d5367 (127.23.100.129:37673): tablet 0ec5bc6523954763a1848a6a84efbdf4 (table test-table [id=0cdf076a797d4961b55a46cb88991a5d]) successfully deleted
I20260806 22:41:09.279597 27287 ts_tablet_manager.cc:1918] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 22:41:09.279739 27287 log.cc:1196] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/0ec5bc6523954763a1848a6a84efbdf4
I20260806 22:41:09.280190 27287 ts_tablet_manager.cc:1939] T 0ec5bc6523954763a1848a6a84efbdf4 P 9cddf5c22aad441eb25ba3e9457a8053: Deleting consensus metadata
I20260806 22:41:09.281352 26355 catalog_manager.cc:4841] TS 9cddf5c22aad441eb25ba3e9457a8053 (127.23.100.130:41031): tablet 0ec5bc6523954763a1848a6a84efbdf4 (table test-table [id=0cdf076a797d4961b55a46cb88991a5d]) successfully deleted
Aug 06 22:41:09 dist-test-slave-zr1t krb5kdc[26314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:41:09.289000 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:09.260059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.23.100.129:37673)
0806 22:41:09.266214 (+ 6155us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:09.266223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:09.266878 (+ 655us) server_negotiation.cc:407] Connection header received
0806 22:41:09.267152 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:09.267161 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:09.267274 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:09.267421 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:09.269829 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:09.271399 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:09.273316 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:09.273966 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:09.274091 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:09.282640 (+ 8549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:09.282658 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:09.282676 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:09.282720 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:09.286126 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:09.286943 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:09.286951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:09.286954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:09.287102 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:09.287462 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:09.287465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:09.287467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:09.287866 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:09.288096 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:09.288520 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 22:41:09.288719 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5963,"thread_start_us":5820,"threads_started":1}
W20260806 22:41:09.289955 26655 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60538
W20260806 22:41:09.290663 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 26473: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:41:09.290918 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 26473
I20260806 22:41:09.319365 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 26668
Aug 06 22:41:09 dist-test-slave-zr1t krb5kdc[26314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:41:09.338912 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:09.320543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.23.100.130:41031)
0806 22:41:09.321158 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:09.321163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:09.321180 (+ 17us) server_negotiation.cc:407] Connection header received
0806 22:41:09.321236 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:09.321239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:09.321314 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:09.321411 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:09.323177 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:09.323991 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:09.326087 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:09.326631 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:09.326809 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:09.332303 (+ 5494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:09.332322 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:09.332340 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:09.332382 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:09.335563 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:09.336425 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:09.336436 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:09.336440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:09.336622 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:09.337023 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:09.337028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:09.337031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:09.337502 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:09.337970 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:09.338249 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 22:41:09.338511 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":333,"threads_started":1}
W20260806 22:41:09.339748 26845 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35908
W20260806 22:41:09.340524 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 26668: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:41:09.340687 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 26668
I20260806 22:41:09.378494 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 26859
Aug 06 22:41:09 dist-test-slave-zr1t krb5kdc[26314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:41:09.394577 27292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:09.379348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.23.100.131:32993)
0806 22:41:09.379913 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:09.379918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:09.379935 (+ 17us) server_negotiation.cc:407] Connection header received
0806 22:41:09.379992 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:09.379996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:09.380088 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:09.380198 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:09.381511 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:09.382348 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:09.383566 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:09.383984 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:09.384080 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:09.388920 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:09.388935 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:09.388953 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:09.388988 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:09.391915 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:09.392638 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:09.392644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:09.392647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:09.392778 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:09.393104 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:09.393107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:09.393109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:09.393452 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:09.393751 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:09.394025 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 22:41:09.394198 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":327,"threads_started":1}
W20260806 22:41:09.395301 27055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49550
W20260806 22:41:09.395993 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 26859: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:41:09.396178 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 26859
I20260806 22:41:09.428035 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 26335
I20260806 22:41:09.429368 26428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:09.550112 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 26335
2026-08-06T22:41:09Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9097 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:41:09 dist-test-slave-zr1t krb5kdc[27306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:41:09 dist-test-slave-zr1t krb5kdc[27306](info): set up 2 sockets
Aug 06 22:41:09 dist-test-slave-zr1t krb5kdc[27306](info): commencing operation
krb5kdc: starting...
W20260806 22:41:10.855535 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:41:11 dist-test-slave-zr1t krb5kdc[27306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:41:11.099443 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:37813
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:37813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:11.444870 27322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:11.445254 27322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:11.445436 27322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:11.453028 27322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:11.453189 27322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:11.453311 27322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:11.453413 27322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:11.453500 27322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:11.463587 27322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:37813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:37813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:11.465003 27322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:11.466368 27322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:11.479055 27330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:11.478957 27335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:11.478993 27333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:11.479817 27322 server_base.cc:1034] running on GCE node
I20260806 22:41:11.480716 27322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:11.482539 27322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:11.483863 27322 hybrid_clock.cc:648] HybridClock initialized: now 1786056071483779 us; error 62 us; skew 500 ppm
I20260806 22:41:11.484297 27322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:11 dist-test-slave-zr1t krb5kdc[27306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056071, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:11.487924 27322 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:41:11.489670 27322 webserver.cc:460] Webserver started at http://127.23.100.190:35955/ using document root <none> and password file <none>
I20260806 22:41:11.490470 27322 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:11.490631 27322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:11.490976 27322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:11.494308 27322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "609532a042f64550bf3650f24f882536"
format_stamp: "Formatted at 2026-08-06 22:41:11 on dist-test-slave-zr1t"
server_key: "49fbf1d22fe9aefbc7f8ffc3b8eee95b"
server_key_iv: "87cd53d634f2234f38fbf2ecf6f0aee9"
server_key_version: "encryptionkey@0"
I20260806 22:41:11.495146 27322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "609532a042f64550bf3650f24f882536"
format_stamp: "Formatted at 2026-08-06 22:41:11 on dist-test-slave-zr1t"
server_key: "49fbf1d22fe9aefbc7f8ffc3b8eee95b"
server_key_iv: "87cd53d634f2234f38fbf2ecf6f0aee9"
server_key_version: "encryptionkey@0"
I20260806 22:41:11.500489 27322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:41:11.504242 27347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:11.505707 27322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:11.505901 27322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "609532a042f64550bf3650f24f882536"
format_stamp: "Formatted at 2026-08-06 22:41:11 on dist-test-slave-zr1t"
server_key: "49fbf1d22fe9aefbc7f8ffc3b8eee95b"
server_key_iv: "87cd53d634f2234f38fbf2ecf6f0aee9"
server_key_version: "encryptionkey@0"
I20260806 22:41:11.506119 27322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:11.519135 27322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:11.523610 27322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:11.523911 27322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:11.543735 27322 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:37813
I20260806 22:41:11.543744 27419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:37813 every 8 connection(s)
I20260806 22:41:11.545148 27322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:11.548859 27421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:11.554191 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 27322
I20260806 22:41:11.554467 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:11.555152 23954 external_mini_cluster.cc:1428] Setting key 63d1dbf805c384d1edd2d5e992c4c371
I20260806 22:41:11.561697 27421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536: Bootstrap starting.
I20260806 22:41:11.565322 27421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:11.566394 27421 log.cc:824] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:11.569307 27421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536: No bootstrap required, opened a new log
Aug 06 22:41:11 dist-test-slave-zr1t krb5kdc[27306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:11.579236 27421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609532a042f64550bf3650f24f882536" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37813 } }
I20260806 22:41:11.579704 27421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:11.579864 27421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609532a042f64550bf3650f24f882536, State: Initialized, Role: FOLLOWER
I20260806 22:41:11.580511 27421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609532a042f64550bf3650f24f882536" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37813 } }
I20260806 22:41:11.580775 27421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:11.580950 27421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:11.581146 27421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:11.581163 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:11.557217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.23.100.190:37813)
0806 22:41:11.558249 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:11.558265 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:11.558314 (+ 49us) server_negotiation.cc:407] Connection header received
0806 22:41:11.560759 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:11.560786 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:11.561265 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:11.561720 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:11.563278 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:11.565270 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:11.566935 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:11.567721 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:11.567907 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:11.572456 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:11.572469 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:11.572494 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:11.572546 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:11.576655 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:11.577383 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:11.577392 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:11.577401 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:11.577656 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:11.578041 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:11.578046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:11.578050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:11.578779 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:11.579068 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:11.579449 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 22:41:11.580017 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":513,"threads_started":1}
I20260806 22:41:11.584066 27421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609532a042f64550bf3650f24f882536" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37813 } }
I20260806 22:41:11.584607 27421 leader_election.cc:302] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 609532a042f64550bf3650f24f882536; no voters:
I20260806 22:41:11.585198 27421 leader_election.cc:288] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:11.585345 27429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:11.586125 27429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [term 1 LEADER]: Becoming Leader. State: Replica: 609532a042f64550bf3650f24f882536, State: Running, Role: LEADER
I20260806 22:41:11.586642 27429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609532a042f64550bf3650f24f882536" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37813 } }
I20260806 22:41:11.587473 27421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:11.589781 27431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 609532a042f64550bf3650f24f882536. Latest consensus state: current_term: 1 leader_uuid: "609532a042f64550bf3650f24f882536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609532a042f64550bf3650f24f882536" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37813 } } }
I20260806 22:41:11.590059 27431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:11.589691 27430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "609532a042f64550bf3650f24f882536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609532a042f64550bf3650f24f882536" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37813 } } }
I20260806 22:41:11.590281 27430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:11.597571 27439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:11.602360 27439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:11.611348 27439 catalog_manager.cc:1347] Generated new cluster ID: 89bb496542264ba4b00acd71033555af
I20260806 22:41:11.611485 27439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:11.636644 27439 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:11.638651 27439 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:11.652390 27439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 609532a042f64550bf3650f24f882536: Generated new TSK 0
I20260806 22:41:11.653417 27439 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:41:11.719298 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37813
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:12.079141 27461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:12.079509 27461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:12.079721 27461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:12.088076 27461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:12.088184 27461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:12.088299 27461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:41:12.098613 27461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:12.099857 27461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:12.101148 27461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:12.115854 27468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:12.115821 27471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:12.115751 27469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:12.116837 27461 server_base.cc:1034] running on GCE node
I20260806 22:41:12.117422 27461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:12.118542 27461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:12.119786 27461 hybrid_clock.cc:648] HybridClock initialized: now 1786056072119735 us; error 53 us; skew 500 ppm
I20260806 22:41:12.120126 27461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:12 dist-test-slave-zr1t krb5kdc[27306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056072, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:12.123400 27461 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:41:12.124897 27461 webserver.cc:460] Webserver started at http://127.23.100.129:38681/ using document root <none> and password file <none>
I20260806 22:41:12.125612 27461 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:12.125720 27461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:12.125981 27461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:12.128885 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "209c7534ad1644309ac4bfd4ea2e2dba"
format_stamp: "Formatted at 2026-08-06 22:41:12 on dist-test-slave-zr1t"
server_key: "d2126a1c43ff1bedd377a4396363b118"
server_key_iv: "11d2055cd58faf57512c1d18a6346fe0"
server_key_version: "encryptionkey@0"
I20260806 22:41:12.129719 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "209c7534ad1644309ac4bfd4ea2e2dba"
format_stamp: "Formatted at 2026-08-06 22:41:12 on dist-test-slave-zr1t"
server_key: "d2126a1c43ff1bedd377a4396363b118"
server_key_iv: "11d2055cd58faf57512c1d18a6346fe0"
server_key_version: "encryptionkey@0"
I20260806 22:41:12.134742 27461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 22:41:12.138193 27479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:12.139612 27461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:12.139752 27461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "209c7534ad1644309ac4bfd4ea2e2dba"
format_stamp: "Formatted at 2026-08-06 22:41:12 on dist-test-slave-zr1t"
server_key: "d2126a1c43ff1bedd377a4396363b118"
server_key_iv: "11d2055cd58faf57512c1d18a6346fe0"
server_key_version: "encryptionkey@0"
I20260806 22:41:12.139923 27461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:12.165903 27461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:12.170588 27461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:12.170847 27461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:12.171816 27461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:12.173202 27461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:12.173278 27461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:12.173367 27461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:12.173410 27461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:12.209720 27461 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:45821
I20260806 22:41:12.209887 27630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:45821 every 8 connection(s)
I20260806 22:41:12.211175 27461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:41:12.220294 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 27461
I20260806 22:41:12.220554 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:41:12.221226 23954 external_mini_cluster.cc:1428] Setting key f838403669d531c7f95d8e1349499b32
Aug 06 22:41:12 dist-test-slave-zr1t krb5kdc[27306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056072, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:12.233927 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:12.213956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:42185 (local address 127.23.100.190:37813)
0806 22:41:12.214455 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:12.214460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:12.215440 (+ 980us) server_negotiation.cc:407] Connection header received
0806 22:41:12.217376 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:12.217382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:12.217478 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:12.217635 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:12.219206 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:12.219987 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:12.221830 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:12.222269 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:12.222430 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:12.227106 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:12.227112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:12.227116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:12.227151 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:12.230021 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:12.231068 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:12.231073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:12.231076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:12.231208 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:12.231732 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:12.231735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:12.231737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:12.232149 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:12.232321 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:12.233099 (+ 778us) server_negotiation.cc:299] Negotiation successful
0806 22:41:12.233392 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":239,"threads_started":1}
I20260806 22:41:12.234622 27635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:12.214504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37813 (local address 127.23.100.129:42185)
0806 22:41:12.215224 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:12.215268 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:12.217082 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:12.217894 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:12.217903 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:12.218438 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:12.219019 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:12.219036 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:12.220150 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:12.220158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:12.221674 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:12.221687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:12.222532 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:12.222538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:12.222754 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:12.223616 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:12.223648 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:12.226813 (+ 3165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:12.230184 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:12.230196 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:12.230217 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:12.230925 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:12.231326 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:12.231333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:12.231339 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:12.231612 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:12.232298 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:12.232304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:12.232664 (+ 360us) client_negotiation.cc:769] Sending connection context
0806 22:41:12.232918 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 22:41:12.233362 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":287,"threads_started":1}
I20260806 22:41:12.236378 27632 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37813
I20260806 22:41:12.236752 27632 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:12.237713 27632 heartbeater.cc:511] Master 127.23.100.190:37813 requested a full tablet report, sending...
I20260806 22:41:12.240275 27369 ts_manager.cc:194] Registered new tserver with Master: 209c7534ad1644309ac4bfd4ea2e2dba (127.23.100.129:45821)
I20260806 22:41:12.241915 27369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:42185
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:41:12.280738 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37813
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:12.631819 27640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:12.632109 27640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:12.632283 27640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:12.639724 27640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:12.639848 27640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:12.640028 27640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:41:12.650265 27640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:12.651590 27640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:12.652956 27640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:12.666528 27649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:12.666600 27651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:12.666944 27647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:12.667691 27640 server_base.cc:1034] running on GCE node
I20260806 22:41:12.668385 27640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:12.669497 27640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:12.670792 27640 hybrid_clock.cc:648] HybridClock initialized: now 1786056072670744 us; error 54 us; skew 500 ppm
I20260806 22:41:12.671126 27640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:12 dist-test-slave-zr1t krb5kdc[27306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056072, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:12.674388 27640 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:41:12.675997 27640 webserver.cc:460] Webserver started at http://127.23.100.130:41739/ using document root <none> and password file <none>
I20260806 22:41:12.676695 27640 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:12.676843 27640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:12.677165 27640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:12.680130 27640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "782e56b20a2141a6814a0b7d99479770"
format_stamp: "Formatted at 2026-08-06 22:41:12 on dist-test-slave-zr1t"
server_key: "d7f56756cc887f5fabe6ae3e9b9585ef"
server_key_iv: "7c2b5f47a1177fc3d06c451660343e20"
server_key_version: "encryptionkey@0"
I20260806 22:41:12.680912 27640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "782e56b20a2141a6814a0b7d99479770"
format_stamp: "Formatted at 2026-08-06 22:41:12 on dist-test-slave-zr1t"
server_key: "d7f56756cc887f5fabe6ae3e9b9585ef"
server_key_iv: "7c2b5f47a1177fc3d06c451660343e20"
server_key_version: "encryptionkey@0"
I20260806 22:41:12.686472 27640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260806 22:41:12.690541 27660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:12.692065 27640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 22:41:12.692288 27640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "782e56b20a2141a6814a0b7d99479770"
format_stamp: "Formatted at 2026-08-06 22:41:12 on dist-test-slave-zr1t"
server_key: "d7f56756cc887f5fabe6ae3e9b9585ef"
server_key_iv: "7c2b5f47a1177fc3d06c451660343e20"
server_key_version: "encryptionkey@0"
I20260806 22:41:12.692539 27640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:12.713992 27640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:12.718525 27640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:12.718823 27640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:12.719799 27640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:12.721254 27640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:12.721328 27640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:12.721462 27640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:12.721549 27640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:12.757357 27640 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:45161
I20260806 22:41:12.757452 27812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:45161 every 8 connection(s)
I20260806 22:41:12.758734 27640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:41:12.766129 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 27640
I20260806 22:41:12.766340 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:41:12.766788 23954 external_mini_cluster.cc:1428] Setting key fddf4d7ce6a2557581cc8414b1bfafc5
Aug 06 22:41:12 dist-test-slave-zr1t krb5kdc[27306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056072, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:12.782840 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:12.761478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:46391 (local address 127.23.100.190:37813)
0806 22:41:12.762021 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:12.762026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:12.762888 (+ 862us) server_negotiation.cc:407] Connection header received
0806 22:41:12.765072 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:12.765078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:12.765177 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:12.765307 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:12.767152 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:12.768034 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:12.770233 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:12.770791 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:12.770917 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:12.775837 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:12.775844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:12.775847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:12.775886 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:12.779156 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:12.780034 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:12.780045 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:12.780048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:12.780189 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:12.780710 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:12.780714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:12.780716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:12.781013 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:12.781194 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:12.782103 (+ 909us) server_negotiation.cc:299] Negotiation successful
0806 22:41:12.782408 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":285,"threads_started":1}
I20260806 22:41:12.783936 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:12.761953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37813 (local address 127.23.100.130:46391)
0806 22:41:12.762705 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:12.762752 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:12.764755 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:12.765624 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:12.765633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:12.766227 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:12.766955 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:12.766977 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:12.768199 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:12.768213 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:12.770027 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:12.770048 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:12.771073 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:12.771083 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:12.771462 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:12.772329 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:12.772371 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:12.775522 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:12.779328 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:12.779339 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:12.779358 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:12.779873 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:12.780329 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:12.780338 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:12.780342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:12.780563 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:12.781163 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:12.781173 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:12.781660 (+ 487us) client_negotiation.cc:769] Sending connection context
0806 22:41:12.781978 (+ 318us) client_negotiation.cc:240] Negotiation successful
0806 22:41:12.782631 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":313,"threads_started":1}
I20260806 22:41:12.785642 27813 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37813
I20260806 22:41:12.786016 27813 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:12.787016 27813 heartbeater.cc:511] Master 127.23.100.190:37813 requested a full tablet report, sending...
I20260806 22:41:12.788841 27369 ts_manager.cc:194] Registered new tserver with Master: 782e56b20a2141a6814a0b7d99479770 (127.23.100.130:45161)
I20260806 22:41:12.789908 27369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:46391
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:41:12.823284 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37813
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:13.175951 27821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:13.176259 27821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:13.176414 27821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:13.183868 27821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:13.184001 27821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:13.184198 27821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:41:13.195045 27821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:13.196525 27821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:13.198233 27821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:13.212196 27829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:13.212420 27833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:13.212921 27826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:13.213630 27821 server_base.cc:1034] running on GCE node
I20260806 22:41:13.214346 27821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:13.215567 27821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:13.216817 27821 hybrid_clock.cc:648] HybridClock initialized: now 1786056073216751 us; error 67 us; skew 500 ppm
I20260806 22:41:13.217123 27821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:13 dist-test-slave-zr1t krb5kdc[27306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056073, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:13.220554 27821 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:41:13.222178 27821 webserver.cc:460] Webserver started at http://127.23.100.131:32855/ using document root <none> and password file <none>
I20260806 22:41:13.222971 27821 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:13.223129 27821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:13.223495 27821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:13.227216 27821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "7fa1a373c25a4a1b98f0814b512b23d7"
format_stamp: "Formatted at 2026-08-06 22:41:13 on dist-test-slave-zr1t"
server_key: "e3a9a2f51629216d5e58d35376e0aea8"
server_key_iv: "e901f69849cb7cd353eb1393e1ea0afa"
server_key_version: "encryptionkey@0"
I20260806 22:41:13.228102 27821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "7fa1a373c25a4a1b98f0814b512b23d7"
format_stamp: "Formatted at 2026-08-06 22:41:13 on dist-test-slave-zr1t"
server_key: "e3a9a2f51629216d5e58d35376e0aea8"
server_key_iv: "e901f69849cb7cd353eb1393e1ea0afa"
server_key_version: "encryptionkey@0"
I20260806 22:41:13.233608 27821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 22:41:13.237391 27846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:13.238869 27821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 22:41:13.239078 27821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "7fa1a373c25a4a1b98f0814b512b23d7"
format_stamp: "Formatted at 2026-08-06 22:41:13 on dist-test-slave-zr1t"
server_key: "e3a9a2f51629216d5e58d35376e0aea8"
server_key_iv: "e901f69849cb7cd353eb1393e1ea0afa"
server_key_version: "encryptionkey@0"
I20260806 22:41:13.239324 27821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:13.246383 27632 heartbeater.cc:503] Master 127.23.100.190:37813 was elected leader, sending a full tablet report...
I20260806 22:41:13.267796 27821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:13.272734 27821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:13.273073 27821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:13.274369 27821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:13.275877 27821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:13.276000 27821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:13.276160 27821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:13.276240 27821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:13.314181 27821 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:33843
I20260806 22:41:13.314245 28003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:33843 every 8 connection(s)
I20260806 22:41:13.315536 27821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:41:13.319164 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 27821
I20260806 22:41:13.319383 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:41:13.319880 23954 external_mini_cluster.cc:1428] Setting key c98388df3c030b477472f9795cca8482
Aug 06 22:41:13 dist-test-slave-zr1t krb5kdc[27306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056073, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:13.337193 28009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:13.318220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:39681 (local address 127.23.100.190:37813)
0806 22:41:13.318678 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:13.318683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:13.319814 (+ 1131us) server_negotiation.cc:407] Connection header received
0806 22:41:13.322326 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:13.322336 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:13.322463 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:13.322590 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:13.324224 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:13.325077 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:13.326755 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:13.327224 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:13.327338 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:13.330972 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:13.330977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:13.330981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:13.331015 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:13.333938 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:13.334755 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:13.334761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:13.334764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:13.334940 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:13.335466 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:13.335470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:13.335472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:13.335828 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:13.335981 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:13.336734 (+ 753us) server_negotiation.cc:299] Negotiation successful
0806 22:41:13.336914 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260806 22:41:13.338289 28010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:13.318799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37813 (local address 127.23.100.131:39681)
0806 22:41:13.319628 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:13.319674 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:13.321717 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:13.322798 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:13.322806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:13.323485 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:13.324050 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:13.324068 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:13.325221 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:13.325225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:13.326592 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:13.326605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:13.327423 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:13.327428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:13.327620 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:13.328285 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:13.328307 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:13.330698 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:13.334095 (+ 3397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:13.334107 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:13.334124 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:13.334562 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:13.335082 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:13.335086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:13.335089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:13.335315 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:13.335990 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:13.335999 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:13.336365 (+ 366us) client_negotiation.cc:769] Sending connection context
0806 22:41:13.336625 (+ 260us) client_negotiation.cc:240] Negotiation successful
0806 22:41:13.337043 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":355,"threads_started":1}
I20260806 22:41:13.339663 28005 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37813
I20260806 22:41:13.340003 28005 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:13.340751 28005 heartbeater.cc:511] Master 127.23.100.190:37813 requested a full tablet report, sending...
I20260806 22:41:13.342278 27369 ts_manager.cc:194] Registered new tserver with Master: 7fa1a373c25a4a1b98f0814b512b23d7 (127.23.100.131:33843)
I20260806 22:41:13.343158 27369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:39681
I20260806 22:41:13.348378 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 22:41:13.388562 28009 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:41:13.385745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.23.100.190:37813)
0806 22:41:13.385928 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:13.385933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:13.386518 (+ 585us) server_negotiation.cc:407] Connection header received
0806 22:41:13.386673 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:13.386677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:13.386743 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:13.386840 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:13.388241 (+ 1401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38248: BlockingRecv error: recv got EOF from 127.0.0.1:38248 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 22:41:13.392195 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 27461
W20260806 22:41:13.395100 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 27461: Not authorized: Client connection negotiation failed: client connection to 127.23.100.129:45821: server requires authentication, but client does not have Kerberos credentials available
I20260806 22:41:13.395318 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 27461
I20260806 22:41:13.417725 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 27640
W20260806 22:41:13.420413 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 27640: Not authorized: Client connection negotiation failed: client connection to 127.23.100.130:45161: server requires authentication, but client does not have Kerberos credentials available
I20260806 22:41:13.420593 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 27640
I20260806 22:41:13.443292 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 27821
W20260806 22:41:13.446902 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 27821: Not authorized: Client connection negotiation failed: client connection to 127.23.100.131:33843: server requires authentication, but client does not have Kerberos credentials available
I20260806 22:41:13.447052 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 27821
I20260806 22:41:13.470549 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 27322
I20260806 22:41:13.472136 27412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:13.577126 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 27322
2026-08-06T22:41:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4023 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:41:13 dist-test-slave-zr1t krb5kdc[28034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:41:13 dist-test-slave-zr1t krb5kdc[28034](info): set up 2 sockets
Aug 06 22:41:13 dist-test-slave-zr1t krb5kdc[28034](info): commencing operation
krb5kdc: starting...
W20260806 22:41:14.882107 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:41:15 dist-test-slave-zr1t krb5kdc[28034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:41:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:41:15.132944 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:46835
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:46835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:15.473685 28055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:15.474057 28055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:15.474208 28055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:15.482024 28055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:15.482189 28055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:15.482296 28055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:15.482405 28055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:15.482486 28055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:15.493005 28055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:46835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:46835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:15.494438 28055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:15.495841 28055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:15.508801 28061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:15.508692 28065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:15.508813 28063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:15.509830 28055 server_base.cc:1034] running on GCE node
I20260806 22:41:15.510771 28055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:15.512279 28055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:15.513572 28055 hybrid_clock.cc:648] HybridClock initialized: now 1786056075513482 us; error 45 us; skew 500 ppm
I20260806 22:41:15.513981 28055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:15 dist-test-slave-zr1t krb5kdc[28034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056075, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:15.522204 28055 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:41:15.523865 28055 webserver.cc:460] Webserver started at http://127.23.100.190:43249/ using document root <none> and password file <none>
I20260806 22:41:15.524600 28055 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:15.524716 28055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:15.525080 28055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:15.528479 28055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "e9e8e45eb07544b083b7d39797df808e"
format_stamp: "Formatted at 2026-08-06 22:41:15 on dist-test-slave-zr1t"
server_key: "3c2a8bef2f52b8307e70e9b749c2b47c"
server_key_iv: "283c192b10430828831a9ab1db569cf3"
server_key_version: "encryptionkey@0"
I20260806 22:41:15.529327 28055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "e9e8e45eb07544b083b7d39797df808e"
format_stamp: "Formatted at 2026-08-06 22:41:15 on dist-test-slave-zr1t"
server_key: "3c2a8bef2f52b8307e70e9b749c2b47c"
server_key_iv: "283c192b10430828831a9ab1db569cf3"
server_key_version: "encryptionkey@0"
I20260806 22:41:15.535012 28055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:41:15.538792 28073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:15.540288 28055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:41:15.540510 28055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "e9e8e45eb07544b083b7d39797df808e"
format_stamp: "Formatted at 2026-08-06 22:41:15 on dist-test-slave-zr1t"
server_key: "3c2a8bef2f52b8307e70e9b749c2b47c"
server_key_iv: "283c192b10430828831a9ab1db569cf3"
server_key_version: "encryptionkey@0"
I20260806 22:41:15.540710 28055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:15.560849 28055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:15.565474 28055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:15.565847 28055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:15.586092 28055 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:46835
I20260806 22:41:15.586117 28147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:46835 every 8 connection(s)
I20260806 22:41:15.587555 28055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:15.591269 28148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:15.597791 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 28055
I20260806 22:41:15.598068 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:15.598891 23954 external_mini_cluster.cc:1428] Setting key 1600a1c50578921a545ac39d63e89e56
I20260806 22:41:15.602574 28148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e: Bootstrap starting.
I20260806 22:41:15.605736 28148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:15.607132 28148 log.cc:824] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:15.610908 28148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e: No bootstrap required, opened a new log
Aug 06 22:41:15 dist-test-slave-zr1t krb5kdc[28034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:15.621963 28148 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e8e45eb07544b083b7d39797df808e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46835 } }
I20260806 22:41:15.622383 28148 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:15.622538 28148 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e8e45eb07544b083b7d39797df808e, State: Initialized, Role: FOLLOWER
I20260806 22:41:15.623242 28148 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e8e45eb07544b083b7d39797df808e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46835 } }
I20260806 22:41:15.623584 28148 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:15.623737 28148 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:15.623956 28148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:15.626906 28148 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e8e45eb07544b083b7d39797df808e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46835 } }
I20260806 22:41:15.627440 28148 leader_election.cc:302] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9e8e45eb07544b083b7d39797df808e; no voters:
I20260806 22:41:15.628772 28148 leader_election.cc:288] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:15.629014 28153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:15.632956 28148 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:15.633082 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:15.600961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.23.100.190:46835)
0806 22:41:15.601800 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:15.601812 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:15.601855 (+ 43us) server_negotiation.cc:407] Connection header received
0806 22:41:15.603845 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:15.603875 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:15.604283 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:15.604715 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:15.606793 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:15.609446 (+ 2653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:15.611404 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:15.612459 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:15.612657 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:15.618058 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:15.618078 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:15.618108 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:15.618171 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:15.623696 (+ 5525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:15.624784 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:15.624797 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:15.624809 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:15.625114 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:15.625606 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:15.625612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:15.625617 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:15.626556 (+ 939us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:15.626973 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:15.631196 (+ 4223us) server_negotiation.cc:299] Negotiation successful
0806 22:41:15.631715 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":359,"threads_started":1}
I20260806 22:41:15.636231 28153 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [term 1 LEADER]: Becoming Leader. State: Replica: e9e8e45eb07544b083b7d39797df808e, State: Running, Role: LEADER
I20260806 22:41:15.636921 28153 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e8e45eb07544b083b7d39797df808e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46835 } }
I20260806 22:41:15.639904 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:41:15.640038 28155 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9e8e45eb07544b083b7d39797df808e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e8e45eb07544b083b7d39797df808e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46835 } } }
I20260806 22:41:15.640511 28155 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:15.640952 28156 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9e8e45eb07544b083b7d39797df808e. Latest consensus state: current_term: 1 leader_uuid: "e9e8e45eb07544b083b7d39797df808e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e8e45eb07544b083b7d39797df808e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46835 } } }
I20260806 22:41:15.641258 28156 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:15.643281 28161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:15.661863 28161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:15.673668 28161 catalog_manager.cc:1347] Generated new cluster ID: dfe479c1b6114660b817610222b0abf5
I20260806 22:41:15.673803 28161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:15.702173 28161 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:15.703593 28161 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:15.718056 28161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9e8e45eb07544b083b7d39797df808e: Generated new TSK 0
I20260806 22:41:15.719301 28161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:41:15.719668 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:15.708976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.23.100.190:46835)
0806 22:41:15.709160 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:15.709166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:15.709848 (+ 682us) server_negotiation.cc:407] Connection header received
0806 22:41:15.710155 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:15.710162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:15.710253 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:15.710329 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:15.711894 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:15.713096 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:15.715139 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:15.715806 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:15.715921 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:15.717702 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:15.717709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:15.717713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:15.718640 (+ 927us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:15.718709 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:15.718741 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:15.718895 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:15.719065 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 22:41:15.719176 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 22:41:15.733489 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 28055
I20260806 22:41:15.735040 28139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:15.828979 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 28055
2026-08-06T22:41:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2246 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:41:15 dist-test-slave-zr1t krb5kdc[28201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:41:15 dist-test-slave-zr1t krb5kdc[28201](info): set up 2 sockets
Aug 06 22:41:15 dist-test-slave-zr1t krb5kdc[28201](info): commencing operation
krb5kdc: starting...
W20260806 22:41:17.162137 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.289s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:41:17 dist-test-slave-zr1t krb5kdc[28201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:41:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:41:17.407295 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:37783
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:37783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:17.748194 28217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:17.748495 28217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:17.748669 28217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:17.755960 28217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:17.756068 28217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:17.756187 28217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:17.756276 28217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:17.756350 28217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:17.766288 28217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:37783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:37783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:17.767591 28217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:17.768997 28217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:17.780944 28229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:17.781042 28226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:17.781068 28227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:17.781891 28217 server_base.cc:1034] running on GCE node
I20260806 22:41:17.782765 28217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:17.784361 28217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:17.785626 28217 hybrid_clock.cc:648] HybridClock initialized: now 1786056077785581 us; error 43 us; skew 500 ppm
I20260806 22:41:17.786003 28217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:17 dist-test-slave-zr1t krb5kdc[28201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056077, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:17.789206 28217 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:41:17.790840 28217 webserver.cc:460] Webserver started at http://127.23.100.190:35179/ using document root <none> and password file <none>
I20260806 22:41:17.791515 28217 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:17.791659 28217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:17.791968 28217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:17.794991 28217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "5365abe814a8408db309ceb39cc5103e"
format_stamp: "Formatted at 2026-08-06 22:41:17 on dist-test-slave-zr1t"
server_key: "2590deef89edaf4d3f001682493963c1"
server_key_iv: "8fa716461bd3b437f68c21db12c11412"
server_key_version: "encryptionkey@0"
I20260806 22:41:17.795761 28217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "5365abe814a8408db309ceb39cc5103e"
format_stamp: "Formatted at 2026-08-06 22:41:17 on dist-test-slave-zr1t"
server_key: "2590deef89edaf4d3f001682493963c1"
server_key_iv: "8fa716461bd3b437f68c21db12c11412"
server_key_version: "encryptionkey@0"
I20260806 22:41:17.801036 28217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 22:41:17.804689 28242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:17.806365 28217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:17.806591 28217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "5365abe814a8408db309ceb39cc5103e"
format_stamp: "Formatted at 2026-08-06 22:41:17 on dist-test-slave-zr1t"
server_key: "2590deef89edaf4d3f001682493963c1"
server_key_iv: "8fa716461bd3b437f68c21db12c11412"
server_key_version: "encryptionkey@0"
I20260806 22:41:17.806831 28217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:17.822278 28217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:17.826583 28217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:17.826890 28217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:17.846370 28217 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:37783
I20260806 22:41:17.846421 28314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:37783 every 8 connection(s)
I20260806 22:41:17.847751 28217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:17.851089 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 28217
I20260806 22:41:17.851323 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:17.851797 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:17.852174 23954 external_mini_cluster.cc:1428] Setting key 0fbaf4c5a3c78567152a3ca8631349eb
I20260806 22:41:17.864686 28318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e: Bootstrap starting.
Aug 06 22:41:17 dist-test-slave-zr1t krb5kdc[28201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:17.868342 28318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:17.869693 28318 log.cc:824] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:17.872781 28318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e: No bootstrap required, opened a new log
I20260806 22:41:17.875111 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:17.854283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.23.100.190:37783)
0806 22:41:17.855095 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:17.855106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:17.855144 (+ 38us) server_negotiation.cc:407] Connection header received
0806 22:41:17.856771 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:17.856793 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:17.857171 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:17.857616 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:17.859042 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:17.860289 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:17.861597 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:17.862334 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:17.862569 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:17.866976 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:17.866985 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:17.867000 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:17.867041 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:17.870613 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:17.871534 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:17.871542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:17.871548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:17.871721 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:17.872141 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:17.872145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:17.872147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:17.872780 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:17.873125 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:17.873594 (+ 469us) server_negotiation.cc:299] Negotiation successful
0806 22:41:17.874071 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":348,"threads_started":1}
I20260806 22:41:17.881392 28318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5365abe814a8408db309ceb39cc5103e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } }
I20260806 22:41:17.881850 28318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:17.881968 28318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5365abe814a8408db309ceb39cc5103e, State: Initialized, Role: FOLLOWER
I20260806 22:41:17.882527 28318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5365abe814a8408db309ceb39cc5103e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } }
I20260806 22:41:17.882795 28318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:17.882941 28318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:17.883126 28318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:17.885946 28318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5365abe814a8408db309ceb39cc5103e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } }
I20260806 22:41:17.886466 28318 leader_election.cc:302] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5365abe814a8408db309ceb39cc5103e; no voters:
I20260806 22:41:17.887027 28318 leader_election.cc:288] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:17.887199 28326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:17.888373 28326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [term 1 LEADER]: Becoming Leader. State: Replica: 5365abe814a8408db309ceb39cc5103e, State: Running, Role: LEADER
I20260806 22:41:17.888911 28326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5365abe814a8408db309ceb39cc5103e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } }
I20260806 22:41:17.889261 28318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:17.892017 28327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5365abe814a8408db309ceb39cc5103e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5365abe814a8408db309ceb39cc5103e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } } }
I20260806 22:41:17.892428 28327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:17.892870 28329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5365abe814a8408db309ceb39cc5103e. Latest consensus state: current_term: 1 leader_uuid: "5365abe814a8408db309ceb39cc5103e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5365abe814a8408db309ceb39cc5103e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } } }
I20260806 22:41:17.893215 28329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:17.893415 28342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:17.898140 28342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:17.906298 28342 catalog_manager.cc:1347] Generated new cluster ID: 4afecc343da24ee2a1d6311c92f5f151
I20260806 22:41:17.906433 28342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:17.930332 28342 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:17.931721 28342 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:17.938359 28342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5365abe814a8408db309ceb39cc5103e: Generated new TSK 0
I20260806 22:41:17.939181 28342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:41:18.004406 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:18.346385 28359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:18.346692 28359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:18.346877 28359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:18.354826 28359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:18.354935 28359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:18.355134 28359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:41:18.364960 28359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:18.366338 28359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:18.367676 28359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:18.380883 28366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:18.380926 28372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:18.380949 28367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:18.381826 28359 server_base.cc:1034] running on GCE node
I20260806 22:41:18.382478 28359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:18.383443 28359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:18.384644 28359 hybrid_clock.cc:648] HybridClock initialized: now 1786056078384582 us; error 63 us; skew 500 ppm
I20260806 22:41:18.384951 28359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:18 dist-test-slave-zr1t krb5kdc[28201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056078, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:18.388325 28359 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:41:18.389964 28359 webserver.cc:460] Webserver started at http://127.23.100.129:42851/ using document root <none> and password file <none>
I20260806 22:41:18.390657 28359 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:18.390792 28359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:18.391109 28359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:18.394146 28359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "b23a0cc9d74e4c5881c61ee98557577f"
format_stamp: "Formatted at 2026-08-06 22:41:18 on dist-test-slave-zr1t"
server_key: "d831209c31f27707c8a2a4a341208a6f"
server_key_iv: "d06e953a6ed04b7d19bde7db7aaa6791"
server_key_version: "encryptionkey@0"
I20260806 22:41:18.394887 28359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "b23a0cc9d74e4c5881c61ee98557577f"
format_stamp: "Formatted at 2026-08-06 22:41:18 on dist-test-slave-zr1t"
server_key: "d831209c31f27707c8a2a4a341208a6f"
server_key_iv: "d06e953a6ed04b7d19bde7db7aaa6791"
server_key_version: "encryptionkey@0"
I20260806 22:41:18.400007 28359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:41:18.403367 28385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:18.404673 28359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 22:41:18.404850 28359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "b23a0cc9d74e4c5881c61ee98557577f"
format_stamp: "Formatted at 2026-08-06 22:41:18 on dist-test-slave-zr1t"
server_key: "d831209c31f27707c8a2a4a341208a6f"
server_key_iv: "d06e953a6ed04b7d19bde7db7aaa6791"
server_key_version: "encryptionkey@0"
I20260806 22:41:18.405043 28359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:18.422873 28359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:18.427462 28359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:18.427799 28359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:18.428855 28359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:18.430343 28359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:18.430449 28359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:18.430629 28359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:18.430701 28359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:18.468190 28359 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:37233
I20260806 22:41:18.468228 28550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:37233 every 8 connection(s)
I20260806 22:41:18.469637 28359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:41:18.479184 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 28359
I20260806 22:41:18.479408 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:41:18.480085 23954 external_mini_cluster.cc:1428] Setting key f21b0ab61bd85d2de2888e896b0aa045
Aug 06 22:41:18 dist-test-slave-zr1t krb5kdc[28201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056078, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:18.490253 28556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:18.472415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:40207 (local address 127.23.100.190:37783)
0806 22:41:18.472890 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:18.472895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:18.473905 (+ 1010us) server_negotiation.cc:407] Connection header received
0806 22:41:18.475751 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:18.475757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:18.475848 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:18.475969 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:18.477489 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:18.478281 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:18.480003 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:18.480454 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:18.480597 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:18.484369 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:18.484374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:18.484378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:18.484409 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:18.487078 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:18.487818 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:18.487826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:18.487829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:18.487964 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:18.488398 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:18.488404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:18.488408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:18.488679 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:18.488836 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:18.489629 (+ 793us) server_negotiation.cc:299] Negotiation successful
0806 22:41:18.489870 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":263,"threads_started":1}
I20260806 22:41:18.491321 28557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:18.472957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37783 (local address 127.23.100.129:40207)
0806 22:41:18.473753 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:18.473793 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:18.475479 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:18.476211 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:18.476219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:18.476726 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:18.477301 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:18.477319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:18.478449 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:18.478454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:18.479837 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:18.479850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:18.480690 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:18.480700 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:18.481034 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:18.481830 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:18.481864 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:18.484136 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:18.487242 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:18.487251 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:18.487267 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:18.487696 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:18.488075 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:18.488080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:18.488083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:18.488284 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:18.488811 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:18.488819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:18.489194 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 22:41:18.489493 (+ 299us) client_negotiation.cc:240] Negotiation successful
0806 22:41:18.490111 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":292,"threads_started":1}
I20260806 22:41:18.493093 28554 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37783
I20260806 22:41:18.493446 28554 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:18.494238 28554 heartbeater.cc:511] Master 127.23.100.190:37783 requested a full tablet report, sending...
I20260806 22:41:18.496703 28265 ts_manager.cc:194] Registered new tserver with Master: b23a0cc9d74e4c5881c61ee98557577f (127.23.100.129:37233)
I20260806 22:41:18.498943 28265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:40207
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:41:18.534498 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:18.891309 28560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:18.891594 28560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:18.891750 28560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:18.899379 28560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:18.899492 28560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:18.899696 28560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:41:18.910039 28560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:18.911451 28560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:18.912774 28560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:18.926046 28569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:18.926090 28571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:18.926115 28567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:18.927170 28560 server_base.cc:1034] running on GCE node
I20260806 22:41:18.927831 28560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:18.928781 28560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:18.929992 28560 hybrid_clock.cc:648] HybridClock initialized: now 1786056078929921 us; error 54 us; skew 500 ppm
I20260806 22:41:18.930326 28560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:18 dist-test-slave-zr1t krb5kdc[28201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056078, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:18.933713 28560 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:41:18.935266 28560 webserver.cc:460] Webserver started at http://127.23.100.130:37041/ using document root <none> and password file <none>
I20260806 22:41:18.935976 28560 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:18.936105 28560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:18.936434 28560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:18.939574 28560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "d9964d89f2324466aa09fa7797a8abd3"
format_stamp: "Formatted at 2026-08-06 22:41:18 on dist-test-slave-zr1t"
server_key: "bacb62d2f64856a74f9dc639455224fa"
server_key_iv: "40ae6ad12d935bc859a9ff28ac8d78f9"
server_key_version: "encryptionkey@0"
I20260806 22:41:18.940376 28560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "d9964d89f2324466aa09fa7797a8abd3"
format_stamp: "Formatted at 2026-08-06 22:41:18 on dist-test-slave-zr1t"
server_key: "bacb62d2f64856a74f9dc639455224fa"
server_key_iv: "40ae6ad12d935bc859a9ff28ac8d78f9"
server_key_version: "encryptionkey@0"
I20260806 22:41:18.945964 28560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 22:41:18.949869 28584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:18.951303 28560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:18.951519 28560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "d9964d89f2324466aa09fa7797a8abd3"
format_stamp: "Formatted at 2026-08-06 22:41:18 on dist-test-slave-zr1t"
server_key: "bacb62d2f64856a74f9dc639455224fa"
server_key_iv: "40ae6ad12d935bc859a9ff28ac8d78f9"
server_key_version: "encryptionkey@0"
I20260806 22:41:18.951735 28560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:18.970952 28560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:18.975716 28560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:18.976037 28560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:18.977057 28560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:18.978492 28560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:18.978567 28560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:18.978696 28560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:18.978751 28560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:19.013924 28560 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:37607
I20260806 22:41:19.014019 28740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:37607 every 8 connection(s)
I20260806 22:41:19.015268 28560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:41:19.021052 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 28560
I20260806 22:41:19.021248 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:41:19.021862 23954 external_mini_cluster.cc:1428] Setting key 90e148f8dc627c8d65b7ec136f780ed0
Aug 06 22:41:19 dist-test-slave-zr1t krb5kdc[28201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056078, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:19.039386 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:19.018033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:59975 (local address 127.23.100.190:37783)
0806 22:41:19.018612 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:19.018617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:19.019356 (+ 739us) server_negotiation.cc:407] Connection header received
0806 22:41:19.021594 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:19.021601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:19.021681 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:19.021780 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:19.023689 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.024478 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.027080 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.027507 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.027604 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:19.032220 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:19.032226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:19.032230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:19.032266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:19.035644 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.036495 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.036504 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.036510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.036678 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.037129 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.037136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.037142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.037626 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:19.037827 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:19.038589 (+ 762us) server_negotiation.cc:299] Negotiation successful
0806 22:41:19.038865 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":383,"threads_started":1}
I20260806 22:41:19.040092 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:19.018476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37783 (local address 127.23.100.130:59975)
0806 22:41:19.019188 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:19.019225 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:19.021208 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:19.022089 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:19.022101 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:19.022839 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:19.023465 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:19.023483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.024638 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.024646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:19.026935 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:19.026956 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.027735 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.027746 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:19.028098 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:19.028877 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:19.028928 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:19.031954 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:19.035814 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:19.035830 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:19.035846 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:19.036357 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:19.036796 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:19.036800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:19.036802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:19.037014 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:19.037822 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:19.037829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:19.038192 (+ 363us) client_negotiation.cc:769] Sending connection context
0806 22:41:19.038470 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 22:41:19.038900 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":281,"threads_started":1}
I20260806 22:41:19.041805 28741 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37783
I20260806 22:41:19.042145 28741 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:19.043175 28741 heartbeater.cc:511] Master 127.23.100.190:37783 requested a full tablet report, sending...
I20260806 22:41:19.045171 28265 ts_manager.cc:194] Registered new tserver with Master: d9964d89f2324466aa09fa7797a8abd3 (127.23.100.130:37607)
I20260806 22:41:19.046545 28265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:59975
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:41:19.078734 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:45519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:19.422331 28749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:19.422693 28749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:19.422861 28749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:19.430337 28749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:19.430506 28749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:19.430687 28749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:41:19.441088 28749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:45519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:19.442785 28749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:19.444247 28749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:19.458258 28761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:19.458345 28759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:19.458228 28764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:19.464707 28749 server_base.cc:1034] running on GCE node
I20260806 22:41:19.465474 28749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:19.466634 28749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:19.467919 28749 hybrid_clock.cc:648] HybridClock initialized: now 1786056079467872 us; error 64 us; skew 500 ppm
I20260806 22:41:19.468345 28749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:19 dist-test-slave-zr1t krb5kdc[28201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056079, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:19.473024 28749 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:41:19.474929 28749 webserver.cc:460] Webserver started at http://127.23.100.131:42251/ using document root <none> and password file <none>
I20260806 22:41:19.475780 28749 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:19.475905 28749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:19.476174 28749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:19.479382 28749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "bc936cf593364c87bbba76d0d1fffb06"
format_stamp: "Formatted at 2026-08-06 22:41:19 on dist-test-slave-zr1t"
server_key: "ecc98c83ac83b4e808f6eb8a9a62e6db"
server_key_iv: "c66321b7788cd5e6c7ad05b1839c4a54"
server_key_version: "encryptionkey@0"
I20260806 22:41:19.480237 28749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "bc936cf593364c87bbba76d0d1fffb06"
format_stamp: "Formatted at 2026-08-06 22:41:19 on dist-test-slave-zr1t"
server_key: "ecc98c83ac83b4e808f6eb8a9a62e6db"
server_key_iv: "c66321b7788cd5e6c7ad05b1839c4a54"
server_key_version: "encryptionkey@0"
I20260806 22:41:19.486030 28749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 22:41:19.489765 28775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:19.491266 28749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:19.491411 28749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "bc936cf593364c87bbba76d0d1fffb06"
format_stamp: "Formatted at 2026-08-06 22:41:19 on dist-test-slave-zr1t"
server_key: "ecc98c83ac83b4e808f6eb8a9a62e6db"
server_key_iv: "c66321b7788cd5e6c7ad05b1839c4a54"
server_key_version: "encryptionkey@0"
I20260806 22:41:19.491659 28749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:19.502123 28554 heartbeater.cc:503] Master 127.23.100.190:37783 was elected leader, sending a full tablet report...
I20260806 22:41:19.513496 28749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:19.518114 28749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:19.518424 28749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:19.519436 28749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:19.520961 28749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:19.521064 28749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:19.521174 28749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:19.521225 28749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:19.557636 28749 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:42323
I20260806 22:41:19.557722 28936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:42323 every 8 connection(s)
I20260806 22:41:19.559074 28749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:41:19.563179 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 28749
I20260806 22:41:19.563380 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:41:19.563862 23954 external_mini_cluster.cc:1428] Setting key c6e3a6a986a99ec222dcc1a0b048ccf1
Aug 06 22:41:19 dist-test-slave-zr1t krb5kdc[28201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056079, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:19.580618 28943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:19.561835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:40811 (local address 127.23.100.190:37783)
0806 22:41:19.562393 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:19.562399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:19.563209 (+ 810us) server_negotiation.cc:407] Connection header received
0806 22:41:19.565412 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:19.565419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:19.565557 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:19.565697 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:19.567502 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.568374 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.570173 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.570653 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.570802 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:19.574627 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:19.574634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:19.574638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:19.574674 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:19.577387 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.578199 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.578205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.578208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.578429 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.578905 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.578912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.578916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.579286 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:19.579465 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:19.580128 (+ 663us) server_negotiation.cc:299] Negotiation successful
0806 22:41:19.580308 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":270,"threads_started":1}
I20260806 22:41:19.581550 28944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:19.562223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37783 (local address 127.23.100.131:40811)
0806 22:41:19.563023 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:19.563072 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:19.565106 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:19.565971 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:19.565980 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:19.566548 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:19.567314 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:19.567334 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.568553 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.568557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:19.570013 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:19.570027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.570916 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.570925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:19.571185 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:19.571909 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:19.571939 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:19.574362 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:19.577589 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:19.577598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:19.577614 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:19.578075 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:19.578580 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:19.578586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:19.578593 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:19.578781 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:19.579424 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:19.579431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:19.579768 (+ 337us) client_negotiation.cc:769] Sending connection context
0806 22:41:19.580020 (+ 252us) client_negotiation.cc:240] Negotiation successful
0806 22:41:19.580402 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":303,"threads_started":1}
I20260806 22:41:19.583045 28939 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37783
I20260806 22:41:19.583388 28939 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:19.584180 28939 heartbeater.cc:511] Master 127.23.100.190:37783 requested a full tablet report, sending...
I20260806 22:41:19.585731 28265 ts_manager.cc:194] Registered new tserver with Master: bc936cf593364c87bbba76d0d1fffb06 (127.23.100.131:42323)
I20260806 22:41:19.586553 28265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:40811
I20260806 22:41:19.591842 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 22:41:19 dist-test-slave-zr1t krb5kdc[28201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056079, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 22:41:19 dist-test-slave-zr1t krb5kdc[28201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056079, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:19.651957 28943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:19.637024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.23.100.190:37783)
0806 22:41:19.637232 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:19.637241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:19.637711 (+ 470us) server_negotiation.cc:407] Connection header received
0806 22:41:19.637930 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:19.637935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:19.638007 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:19.638116 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:19.639579 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.640327 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.641649 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.642069 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.642208 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:19.646352 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:19.646358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:19.646361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:19.646398 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:19.649088 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.649899 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.649904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.649906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.650043 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.650465 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.650469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.650471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.650813 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:19.651065 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:19.651400 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 22:41:19.651645 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 22:41:19.652535 28265 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40726
I20260806 22:41:19.654024 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 28359
Aug 06 22:41:19 dist-test-slave-zr1t krb5kdc[28201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056079, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:41:19.670934 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:19.655125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53332 (local address 127.23.100.129:37233)
0806 22:41:19.655888 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:19.655895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:19.655918 (+ 23us) server_negotiation.cc:407] Connection header received
0806 22:41:19.655975 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:19.655981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:19.656178 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:19.656386 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:19.657512 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.658744 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.659975 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.660563 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.660706 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:19.664789 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:19.664802 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:19.664819 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:19.664858 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:19.667856 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.668619 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.668627 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.668629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.668772 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.669172 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.669177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.669180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.669647 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:19.669959 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:19.670285 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 22:41:19.670579 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":331,"threads_started":1}
W20260806 22:41:19.672224 28544 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53332
W20260806 22:41:19.673480 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 28359: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:41:19.673792 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 28359
I20260806 22:41:19.701611 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 28560
Aug 06 22:41:19 dist-test-slave-zr1t krb5kdc[28201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056079, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:41:19.719098 28959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:19.702654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.23.100.130:37607)
0806 22:41:19.703328 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:19.703335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:19.703363 (+ 28us) server_negotiation.cc:407] Connection header received
0806 22:41:19.703432 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:19.703439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:19.703649 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:19.703790 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:19.705216 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.706452 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.707776 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.708398 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.708506 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:19.712822 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:19.712835 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:19.712850 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:19.712893 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:19.716059 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.716787 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.716798 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.716807 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.716950 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.717357 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.717361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.717363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.717803 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:19.718153 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:19.718449 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 22:41:19.718737 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":381,"threads_started":1}
W20260806 22:41:19.720351 28733 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43980
W20260806 22:41:19.721427 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 28560: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:41:19.721642 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 28560
I20260806 22:41:19.746750 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 28749
Aug 06 22:41:19 dist-test-slave-zr1t krb5kdc[28201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056079, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:41:19.764951 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:19.747941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.23.100.131:42323)
0806 22:41:19.748598 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:19.748605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:19.748635 (+ 30us) server_negotiation.cc:407] Connection header received
0806 22:41:19.748698 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:19.748704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:19.748941 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:19.749121 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:19.750808 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.751974 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.753246 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:19.753931 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:19.754068 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:19.758961 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:19.758975 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:19.758991 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:19.759036 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:19.762103 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.762792 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.762800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.762802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.762929 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:19.763302 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:19.763307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:19.763314 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:19.763670 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:19.763983 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:19.764295 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 22:41:19.764592 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":334,"threads_started":1}
W20260806 22:41:19.766472 28929 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53000
W20260806 22:41:19.767676 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 28749: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:41:19.767912 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 28749
I20260806 22:41:19.790163 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 28217
I20260806 22:41:19.791520 28306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:19.896926 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 28217
2026-08-06T22:41:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4070 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:41:20 dist-test-slave-zr1t krb5kdc[28965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:41:20 dist-test-slave-zr1t krb5kdc[28965](info): set up 2 sockets
Aug 06 22:41:20 dist-test-slave-zr1t krb5kdc[28965](info): commencing operation
krb5kdc: starting...
W20260806 22:41:21.268992 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.325s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:41:21 dist-test-slave-zr1t krb5kdc[28965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:41:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:41:21.521471 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:32905
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:32905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:21.867856 28983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:21.868160 28983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:21.868328 28983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:21.875597 28983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:21.875715 28983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:21.875808 28983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:21.875918 28983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:21.875977 28983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:21.886150 28983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:32905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:32905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:21.887566 28983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:21.888897 28983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:21.900806 28993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:21.900738 28991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:21.900761 28989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:21.901953 28983 server_base.cc:1034] running on GCE node
I20260806 22:41:21.902776 28983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:21.904232 28983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:21.905460 28983 hybrid_clock.cc:648] HybridClock initialized: now 1786056081905387 us; error 53 us; skew 500 ppm
I20260806 22:41:21.905848 28983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:21 dist-test-slave-zr1t krb5kdc[28965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056081, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:21.914119 28983 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:41:21.915997 28983 webserver.cc:460] Webserver started at http://127.23.100.190:43135/ using document root <none> and password file <none>
I20260806 22:41:21.916797 28983 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:21.916947 28983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:21.917327 28983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:21.920791 28983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "cef0cc41896f45f7b8c8cb5521726d6d"
format_stamp: "Formatted at 2026-08-06 22:41:21 on dist-test-slave-zr1t"
server_key: "c52193022bbf48c93b67f96dfcf01838"
server_key_iv: "707127e3babba861055a51a46b1c737a"
server_key_version: "encryptionkey@0"
I20260806 22:41:21.921814 28983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "cef0cc41896f45f7b8c8cb5521726d6d"
format_stamp: "Formatted at 2026-08-06 22:41:21 on dist-test-slave-zr1t"
server_key: "c52193022bbf48c93b67f96dfcf01838"
server_key_iv: "707127e3babba861055a51a46b1c737a"
server_key_version: "encryptionkey@0"
I20260806 22:41:21.927809 28983 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 22:41:21.931747 29007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:21.933226 28983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:41:21.933452 28983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "cef0cc41896f45f7b8c8cb5521726d6d"
format_stamp: "Formatted at 2026-08-06 22:41:21 on dist-test-slave-zr1t"
server_key: "c52193022bbf48c93b67f96dfcf01838"
server_key_iv: "707127e3babba861055a51a46b1c737a"
server_key_version: "encryptionkey@0"
I20260806 22:41:21.933753 28983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:21.952821 28983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:21.957569 28983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:21.957895 28983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:21.979063 28983 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:32905
I20260806 22:41:21.979090 29072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:32905 every 8 connection(s)
I20260806 22:41:21.980662 28983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:21.984581 29076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:21.986603 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 28983
I20260806 22:41:21.986896 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:21.987634 23954 external_mini_cluster.cc:1428] Setting key ef0bb928019562e3114dd347d6da3212
I20260806 22:41:21.997819 29076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d: Bootstrap starting.
I20260806 22:41:22.001556 29076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:22.002647 29076 log.cc:824] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d: Log is configured to *not* fsync() on all Append() calls
Aug 06 22:41:22 dist-test-slave-zr1t krb5kdc[28965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:22.006278 29076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d: No bootstrap required, opened a new log
I20260806 22:41:22.015446 29082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:21.989948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.23.100.190:32905)
0806 22:41:21.990808 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:21.990825 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:21.990876 (+ 51us) server_negotiation.cc:407] Connection header received
0806 22:41:21.993265 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:21.993295 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:21.993841 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:21.994311 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:21.995892 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:21.997934 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:21.999405 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:22.000454 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:22.000641 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:22.005701 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:22.005715 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:22.005735 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:22.005806 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:22.010271 (+ 4465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:22.011163 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:22.011172 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:22.011182 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:22.011415 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:22.011810 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:22.011816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:22.011819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:22.012641 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:22.012969 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:22.013404 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 22:41:22.013939 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":334,"threads_started":1}
I20260806 22:41:22.017663 29076 raft_consensus.cc:348] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef0cc41896f45f7b8c8cb5521726d6d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 32905 } }
I20260806 22:41:22.018056 29076 raft_consensus.cc:374] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:22.018173 29076 raft_consensus.cc:729] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef0cc41896f45f7b8c8cb5521726d6d, State: Initialized, Role: FOLLOWER
I20260806 22:41:22.018790 29076 consensus_queue.cc:260] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef0cc41896f45f7b8c8cb5521726d6d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 32905 } }
I20260806 22:41:22.019038 29076 raft_consensus.cc:388] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:22.019181 29076 raft_consensus.cc:482] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:22.019349 29076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:22.021622 29076 raft_consensus.cc:504] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef0cc41896f45f7b8c8cb5521726d6d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 32905 } }
I20260806 22:41:22.022161 29076 leader_election.cc:302] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cef0cc41896f45f7b8c8cb5521726d6d; no voters:
I20260806 22:41:22.022945 29076 leader_election.cc:288] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:22.023078 29089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:22.023777 29089 raft_consensus.cc:686] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [term 1 LEADER]: Becoming Leader. State: Replica: cef0cc41896f45f7b8c8cb5521726d6d, State: Running, Role: LEADER
I20260806 22:41:22.024358 29089 consensus_queue.cc:237] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef0cc41896f45f7b8c8cb5521726d6d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 32905 } }
I20260806 22:41:22.025280 29076 sys_catalog.cc:564] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:22.027439 29090 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cef0cc41896f45f7b8c8cb5521726d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef0cc41896f45f7b8c8cb5521726d6d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 32905 } } }
I20260806 22:41:22.027524 29091 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cef0cc41896f45f7b8c8cb5521726d6d. Latest consensus state: current_term: 1 leader_uuid: "cef0cc41896f45f7b8c8cb5521726d6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef0cc41896f45f7b8c8cb5521726d6d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 32905 } } }
I20260806 22:41:22.027930 29090 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:22.027935 29091 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:22.028927 29094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:22.035601 29094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:22.047309 29094 catalog_manager.cc:1347] Generated new cluster ID: 0cd65f11b53c458183b40b95f75d63d8
I20260806 22:41:22.047461 29094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:22.062433 29094 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:22.063758 29094 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:22.073004 29094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cef0cc41896f45f7b8c8cb5521726d6d: Generated new TSK 0
I20260806 22:41:22.073932 29094 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:41:22.146136 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:32905
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:22.534672 29114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:22.535072 29114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:22.535177 29114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:22.543049 29114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:22.543180 29114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:22.543310 29114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:41:22.553651 29114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:32905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:22.555408 29114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:22.556763 29114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:22.571305 29124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:22.571254 29128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:22.571281 29123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:22.572522 29114 server_base.cc:1034] running on GCE node
I20260806 22:41:22.573407 29114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:22.574541 29114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:22.575788 29114 hybrid_clock.cc:648] HybridClock initialized: now 1786056082575742 us; error 66 us; skew 500 ppm
I20260806 22:41:22.576150 29114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:22 dist-test-slave-zr1t krb5kdc[28965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:22.579846 29114 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:41:22.581426 29114 webserver.cc:460] Webserver started at http://127.23.100.129:39015/ using document root <none> and password file <none>
I20260806 22:41:22.582233 29114 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:22.582369 29114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:22.582672 29114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:22.585988 29114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "ac51aa69583546f48a71daea95dab134"
format_stamp: "Formatted at 2026-08-06 22:41:22 on dist-test-slave-zr1t"
server_key: "69b271cc2304b2985251958821e75994"
server_key_iv: "05c40f63e63f7f68bc929ea6d09b37ed"
server_key_version: "encryptionkey@0"
I20260806 22:41:22.586839 29114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "ac51aa69583546f48a71daea95dab134"
format_stamp: "Formatted at 2026-08-06 22:41:22 on dist-test-slave-zr1t"
server_key: "69b271cc2304b2985251958821e75994"
server_key_iv: "05c40f63e63f7f68bc929ea6d09b37ed"
server_key_version: "encryptionkey@0"
I20260806 22:41:22.592792 29114 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 22:41:22.596819 29139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:22.598603 29114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:22.598809 29114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "ac51aa69583546f48a71daea95dab134"
format_stamp: "Formatted at 2026-08-06 22:41:22 on dist-test-slave-zr1t"
server_key: "69b271cc2304b2985251958821e75994"
server_key_iv: "05c40f63e63f7f68bc929ea6d09b37ed"
server_key_version: "encryptionkey@0"
I20260806 22:41:22.599061 29114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:22.619751 29114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:22.624388 29114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:22.624723 29114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:22.625952 29114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:22.627699 29114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:22.627784 29114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:22.627957 29114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:22.628014 29114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:22.664518 29114 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:35917
I20260806 22:41:22.664598 29308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:35917 every 8 connection(s)
I20260806 22:41:22.665923 29114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:41:22.667735 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 29114
I20260806 22:41:22.667979 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:41:22.668768 23954 external_mini_cluster.cc:1428] Setting key 43985be6092e98b2787bbfa20bcd73be
Aug 06 22:41:22 dist-test-slave-zr1t krb5kdc[28965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056082, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:22.689692 29312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:22.668718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:48015 (local address 127.23.100.190:32905)
0806 22:41:22.669238 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:22.669247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:22.670355 (+ 1108us) server_negotiation.cc:407] Connection header received
0806 22:41:22.673052 (+ 2697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:22.673056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:22.673127 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:22.673229 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:22.675521 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:22.676257 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:22.678928 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:22.679417 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:22.679545 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:22.683667 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:22.683673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:22.683676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:22.683712 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:22.686423 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:22.687214 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:22.687231 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:22.687235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:22.687369 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:22.687843 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:22.687847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:22.687849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:22.688137 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:22.688301 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:22.689028 (+ 727us) server_negotiation.cc:299] Negotiation successful
0806 22:41:22.689271 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":259,"threads_started":1}
I20260806 22:41:22.690711 29313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:22.669298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:32905 (local address 127.23.100.129:48015)
0806 22:41:22.670204 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:22.670249 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:22.672735 (+ 2486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:22.673550 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:22.673565 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:22.674425 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:22.675272 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:22.675302 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:22.676440 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:22.676448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:22.678739 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:22.678758 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:22.679684 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:22.679695 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:22.680032 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:22.680849 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:22.680881 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:22.683409 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:22.686577 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:22.686588 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:22.686606 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:22.687074 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:22.687488 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:22.687493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:22.687497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:22.687716 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:22.688273 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:22.688282 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:22.688656 (+ 374us) client_negotiation.cc:769] Sending connection context
0806 22:41:22.688935 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 22:41:22.689505 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":393,"threads_started":1}
I20260806 22:41:22.692361 29310 heartbeater.cc:347] Connected to a master server at 127.23.100.190:32905
I20260806 22:41:22.692682 29310 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:22.693413 29310 heartbeater.cc:511] Master 127.23.100.190:32905 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
I20260806 22:41:22.695626 29030 ts_manager.cc:194] Registered new tserver with Master: ac51aa69583546f48a71daea95dab134 (127.23.100.129:35917)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
I20260806 22:41:22.697999 29030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:48015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:41:22.725306 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:32905
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:23.082185 29318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:23.082516 29318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:23.082613 29318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:23.090061 29318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:23.090178 29318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:23.090305 29318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:41:23.100095 29318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:32905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:23.101424 29318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:23.102838 29318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:23.116986 29329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:23.117096 29325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:23.117065 29326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:23.118042 29318 server_base.cc:1034] running on GCE node
I20260806 22:41:23.118738 29318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:23.119844 29318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:23.121106 29318 hybrid_clock.cc:648] HybridClock initialized: now 1786056083121054 us; error 43 us; skew 500 ppm
I20260806 22:41:23.121472 29318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:23 dist-test-slave-zr1t krb5kdc[28965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056083, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:23.124897 29318 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:41:23.126698 29318 webserver.cc:460] Webserver started at http://127.23.100.130:37045/ using document root <none> and password file <none>
I20260806 22:41:23.127365 29318 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:23.127476 29318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:23.127743 29318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:23.130764 29318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "83a255d8687c48e0a1eedb98958456c8"
format_stamp: "Formatted at 2026-08-06 22:41:23 on dist-test-slave-zr1t"
server_key: "1739e15ff3d16988a934124b7f7bdb2c"
server_key_iv: "a8e94d7f255cff0677b5cf7df4ad67c0"
server_key_version: "encryptionkey@0"
I20260806 22:41:23.131597 29318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "83a255d8687c48e0a1eedb98958456c8"
format_stamp: "Formatted at 2026-08-06 22:41:23 on dist-test-slave-zr1t"
server_key: "1739e15ff3d16988a934124b7f7bdb2c"
server_key_iv: "a8e94d7f255cff0677b5cf7df4ad67c0"
server_key_version: "encryptionkey@0"
I20260806 22:41:23.137182 29318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260806 22:41:23.141007 29340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:23.142768 29318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 22:41:23.142992 29318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "83a255d8687c48e0a1eedb98958456c8"
format_stamp: "Formatted at 2026-08-06 22:41:23 on dist-test-slave-zr1t"
server_key: "1739e15ff3d16988a934124b7f7bdb2c"
server_key_iv: "a8e94d7f255cff0677b5cf7df4ad67c0"
server_key_version: "encryptionkey@0"
I20260806 22:41:23.143242 29318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:23.176403 29318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:23.181712 29318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:23.182094 29318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:23.183223 29318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:23.184727 29318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:23.184813 29318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:23.184988 29318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:23.185051 29318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:23.220247 29318 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:33349
I20260806 22:41:23.220283 29502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:33349 every 8 connection(s)
I20260806 22:41:23.221774 29318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:41:23.232053 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 29318
I20260806 22:41:23.232327 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:41:23.232798 23954 external_mini_cluster.cc:1428] Setting key 3d13cb75d9fb43a2831e38615551f106
Aug 06 22:41:23 dist-test-slave-zr1t krb5kdc[28965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056083, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:23.244421 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:23.224641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:53669 (local address 127.23.100.190:32905)
0806 22:41:23.225123 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:23.225129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:23.226342 (+ 1213us) server_negotiation.cc:407] Connection header received
0806 22:41:23.228427 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:23.228432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:23.228546 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:23.228649 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:23.230379 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.231186 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.233341 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.233875 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.234028 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:23.237815 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:23.237821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:23.237824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:23.237862 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:23.240935 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:23.241887 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:23.241892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:23.241894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:23.242046 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:23.242538 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:23.242542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:23.242544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:23.242846 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:23.243012 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:23.243764 (+ 752us) server_negotiation.cc:299] Negotiation successful
0806 22:41:23.244023 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260806 22:41:23.245937 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:23.225235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:32905 (local address 127.23.100.130:53669)
0806 22:41:23.226163 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:23.226206 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:23.228129 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:23.228900 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:23.228909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:23.229474 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:23.230168 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:23.230186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.231354 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.231360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:23.233131 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:23.233149 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.234138 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.234146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:23.234388 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:23.235110 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:23.235137 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:23.237521 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:23.241120 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:23.241130 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:23.241152 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:23.241733 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:23.242165 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:23.242170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:23.242174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:23.242404 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:23.242982 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:23.242990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:23.243381 (+ 391us) client_negotiation.cc:769] Sending connection context
0806 22:41:23.243657 (+ 276us) client_negotiation.cc:240] Negotiation successful
0806 22:41:23.244269 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":391,"threads_started":1}
I20260806 22:41:23.247637 29503 heartbeater.cc:347] Connected to a master server at 127.23.100.190:32905
I20260806 22:41:23.248013 29503 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:23.248864 29503 heartbeater.cc:511] Master 127.23.100.190:32905 requested a full tablet report, sending...
I20260806 22:41:23.250444 29030 ts_manager.cc:194] Registered new tserver with Master: 83a255d8687c48e0a1eedb98958456c8 (127.23.100.130:33349)
I20260806 22:41:23.251796 29030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:53669
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:41:23.290568 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:32905
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:23.634402 29509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:23.634711 29509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:23.634817 29509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:23.642097 29509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:23.642210 29509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:23.642352 29509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:41:23.652485 29509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:32905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:23.653821 29509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:23.655414 29509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:23.670149 29519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:23.670193 29522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:23.670215 29517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:23.671775 29509 server_base.cc:1034] running on GCE node
I20260806 22:41:23.672341 29509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:23.673240 29509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:23.674460 29509 hybrid_clock.cc:648] HybridClock initialized: now 1786056083674376 us; error 83 us; skew 500 ppm
I20260806 22:41:23.674789 29509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:23 dist-test-slave-zr1t krb5kdc[28965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056083, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:23.678205 29509 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:41:23.679852 29509 webserver.cc:460] Webserver started at http://127.23.100.131:40697/ using document root <none> and password file <none>
I20260806 22:41:23.680570 29509 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:23.680712 29509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:23.681052 29509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:23.684127 29509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "3f02743e65dd4787bfd1d7fbb808ee83"
format_stamp: "Formatted at 2026-08-06 22:41:23 on dist-test-slave-zr1t"
server_key: "0a3e931cbb742869c50ec91059fa2518"
server_key_iv: "70fc144bd775a5c294732a1bb41f633d"
server_key_version: "encryptionkey@0"
I20260806 22:41:23.684930 29509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "3f02743e65dd4787bfd1d7fbb808ee83"
format_stamp: "Formatted at 2026-08-06 22:41:23 on dist-test-slave-zr1t"
server_key: "0a3e931cbb742869c50ec91059fa2518"
server_key_iv: "70fc144bd775a5c294732a1bb41f633d"
server_key_version: "encryptionkey@0"
I20260806 22:41:23.690263 29509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 22:41:23.693854 29533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:23.695165 29509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:41:23.695361 29509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "3f02743e65dd4787bfd1d7fbb808ee83"
format_stamp: "Formatted at 2026-08-06 22:41:23 on dist-test-slave-zr1t"
server_key: "0a3e931cbb742869c50ec91059fa2518"
server_key_iv: "70fc144bd775a5c294732a1bb41f633d"
server_key_version: "encryptionkey@0"
I20260806 22:41:23.695542 29509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:23.701131 29310 heartbeater.cc:503] Master 127.23.100.190:32905 was elected leader, sending a full tablet report...
I20260806 22:41:23.711717 29509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:23.716406 29509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:23.716665 29509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:23.717685 29509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:23.719141 29509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:23.719218 29509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:23.719307 29509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:23.719357 29509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:23.756219 29509 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:44239
I20260806 22:41:23.756253 29690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:44239 every 8 connection(s)
I20260806 22:41:23.757750 29509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:41:23.767191 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 29509
I20260806 22:41:23.767429 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:41:23.767897 23954 external_mini_cluster.cc:1428] Setting key 2014b936915e0243ef24e33a73d00f32
Aug 06 22:41:23 dist-test-slave-zr1t krb5kdc[28965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056083, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:23.778885 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:23.760417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:50651 (local address 127.23.100.190:32905)
0806 22:41:23.760870 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:23.760875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:23.761854 (+ 979us) server_negotiation.cc:407] Connection header received
0806 22:41:23.763784 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:23.763790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:23.763891 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:23.764012 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:23.765615 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.766407 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.768388 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.768974 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.769106 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:23.772957 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:23.772963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:23.772966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:23.773004 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:23.775605 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:23.776360 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:23.776367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:23.776371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:23.776528 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:23.776968 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:23.776975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:23.776980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:23.777256 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:23.777419 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:23.778229 (+ 810us) server_negotiation.cc:299] Negotiation successful
0806 22:41:23.778492 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":244,"threads_started":1}
I20260806 22:41:23.779892 29695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:23.760874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:32905 (local address 127.23.100.131:50651)
0806 22:41:23.761686 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:23.761732 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:23.763495 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:23.764260 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:23.764268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:23.764813 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:23.765404 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:23.765422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.766597 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.766601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:23.768228 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:23.768243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.769217 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.769224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:23.769559 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:23.770356 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:23.770381 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:23.772720 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:23.775756 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:23.775766 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:23.775788 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:23.776234 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:23.776643 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:23.776649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:23.776653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:23.776855 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:23.777388 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:23.777397 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:23.777830 (+ 433us) client_negotiation.cc:769] Sending connection context
0806 22:41:23.778139 (+ 309us) client_negotiation.cc:240] Negotiation successful
0806 22:41:23.778727 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":331,"threads_started":1}
I20260806 22:41:23.781562 29692 heartbeater.cc:347] Connected to a master server at 127.23.100.190:32905
I20260806 22:41:23.781950 29692 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:23.783039 29692 heartbeater.cc:511] Master 127.23.100.190:32905 requested a full tablet report, sending...
I20260806 22:41:23.784780 29030 ts_manager.cc:194] Registered new tserver with Master: 3f02743e65dd4787bfd1d7fbb808ee83 (127.23.100.131:44239)
I20260806 22:41:23.785679 29030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:50651
I20260806 22:41:23.796926 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 22:41:23 dist-test-slave-zr1t krb5kdc[28965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 22:41:23 dist-test-slave-zr1t krb5kdc[28965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:41:23.855180 29701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:23.839255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.23.100.129:35917)
0806 22:41:23.839959 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:23.839967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:23.839992 (+ 25us) server_negotiation.cc:407] Connection header received
0806 22:41:23.840054 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:23.840060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:23.840275 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:23.840444 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:23.841999 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.843049 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.844301 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.844887 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.845042 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:23.849234 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:23.849249 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:23.849264 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:23.849301 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:23.852239 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:23.852934 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:23.852942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:23.852945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:23.853073 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:23.853461 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:23.853470 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:23.853472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:23.853874 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:23.854199 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:23.854523 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 22:41:23.854805 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":371,"threads_started":1}
Aug 06 22:41:23 dist-test-slave-zr1t krb5kdc[28965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:23.874645 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:23.860451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.23.100.190:32905)
0806 22:41:23.860698 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:23.860703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:23.860811 (+ 108us) server_negotiation.cc:407] Connection header received
0806 22:41:23.860990 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:23.860994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:23.861062 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:23.861150 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:23.862829 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.863564 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.864911 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.865299 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.865430 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:23.869442 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:23.869448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:23.869451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:23.869489 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:23.872194 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:23.872781 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:23.872784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:23.872786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:23.872910 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:23.873214 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:23.873217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:23.873219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:23.873643 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:23.873827 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:23.874093 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 22:41:23.874266 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260806 22:41:23.875267 29030 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35010
I20260806 22:41:23.876302 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 29114
I20260806 22:41:23.889612 29701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:23.877212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.23.100.129:35917)
0806 22:41:23.877430 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:23.877438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:23.877765 (+ 327us) server_negotiation.cc:407] Connection header received
0806 22:41:23.877968 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:23.877973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:23.878038 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:23.878137 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:23.879355 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.880100 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.881330 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:23.881877 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:23.882030 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:23.884336 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:23.884341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:23.884344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:23.884379 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:23.886926 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:23.887627 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:23.887634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:23.887638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:23.887777 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:23.888160 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:23.888165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:23.888169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:23.888431 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:23.888682 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:23.888994 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 22:41:23.889241 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 22:41:23.890194 29302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:23.973811 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 29114
I20260806 22:41:24.003857 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 29318
Aug 06 22:41:24 dist-test-slave-zr1t krb5kdc[28965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:41:24.020696 29709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:24.004841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.23.100.130:33349)
0806 22:41:24.005359 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:24.005367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:24.005392 (+ 25us) server_negotiation.cc:407] Connection header received
0806 22:41:24.005453 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:24.005460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:24.005745 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:24.005924 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:24.007195 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:24.008241 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:24.009758 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:24.010311 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:24.010466 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:24.014795 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:24.014808 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:24.014822 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:24.014860 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:24.017851 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:24.018537 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:24.018546 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:24.018548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:24.018675 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:24.019059 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:24.019065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:24.019068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:24.019419 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:24.019746 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:24.020071 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 22:41:24.020341 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":270,"threads_started":1}
I20260806 22:41:24.022046 29498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:24.105980 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 29318
I20260806 22:41:24.134811 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 29509
Aug 06 22:41:24 dist-test-slave-zr1t krb5kdc[28965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:41:24.152558 29711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:24.135891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.23.100.131:44239)
0806 22:41:24.136620 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:24.136627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:24.136662 (+ 35us) server_negotiation.cc:407] Connection header received
0806 22:41:24.136726 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:24.136732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:24.136979 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:24.137201 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:24.138770 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:24.139907 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:24.141224 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:24.141980 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:24.142121 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:24.146725 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:24.146738 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:24.146753 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:24.146791 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:24.149649 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:24.150338 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:24.150353 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:24.150355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:24.150486 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:24.150852 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:24.150857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:24.150860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:24.151221 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:24.151557 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:24.151889 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 22:41:24.152191 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":400,"threads_started":1}
I20260806 22:41:24.154160 29684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:24.238843 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 29509
I20260806 22:41:24.261332 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 28983
I20260806 22:41:24.262732 29068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:24.368076 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 28983
2026-08-06T22:41:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4472 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:41:24 dist-test-slave-zr1t krb5kdc[29720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:41:24 dist-test-slave-zr1t krb5kdc[29720](info): set up 2 sockets
Aug 06 22:41:24 dist-test-slave-zr1t krb5kdc[29720](info): commencing operation
krb5kdc: starting...
W20260806 22:41:25.691507 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:41:25 dist-test-slave-zr1t krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:41:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:41:25.964963 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:45841
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:45841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:26.315733 29739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:26.316092 29739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:26.316267 29739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:26.323750 29739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:26.323902 29739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:26.324018 29739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:26.324119 29739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:26.324206 29739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:26.334280 29739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:45841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:45841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:26.335680 29739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:26.337018 29739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:26.348804 29750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:26.348872 29747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:26.348901 29748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:26.349921 29739 server_base.cc:1034] running on GCE node
I20260806 22:41:26.350795 29739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:26.352238 29739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:26.353456 29739 hybrid_clock.cc:648] HybridClock initialized: now 1786056086353411 us; error 40 us; skew 500 ppm
I20260806 22:41:26.353883 29739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:26 dist-test-slave-zr1t krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056086, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:26.357120 29739 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:41:26.358512 29739 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:26.358660 29739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:26.359021 29739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:26.362195 29739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "39877c5fca8d4448916bea31557a2879"
format_stamp: "Formatted at 2026-08-06 22:41:26 on dist-test-slave-zr1t"
server_key: "473bf621d2b8027adcccdc628981fdb6"
server_key_iv: "0873141352e3087074796b712edac3d9"
server_key_version: "encryptionkey@0"
I20260806 22:41:26.363101 29739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "39877c5fca8d4448916bea31557a2879"
format_stamp: "Formatted at 2026-08-06 22:41:26 on dist-test-slave-zr1t"
server_key: "473bf621d2b8027adcccdc628981fdb6"
server_key_iv: "0873141352e3087074796b712edac3d9"
server_key_version: "encryptionkey@0"
I20260806 22:41:26.368520 29739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 22:41:26.372229 29758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:26.373725 29739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:26.373944 29739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "39877c5fca8d4448916bea31557a2879"
format_stamp: "Formatted at 2026-08-06 22:41:26 on dist-test-slave-zr1t"
server_key: "473bf621d2b8027adcccdc628981fdb6"
server_key_iv: "0873141352e3087074796b712edac3d9"
server_key_version: "encryptionkey@0"
I20260806 22:41:26.374193 29739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:26.391692 29739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:26.396226 29739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:26.396534 29739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:26.415925 29739 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:45841
I20260806 22:41:26.415943 29835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:45841 every 8 connection(s)
I20260806 22:41:26.416976 29739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:26.420148 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 29739
I20260806 22:41:26.420390 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:26.420771 29836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:26.421116 23954 external_mini_cluster.cc:1428] Setting key 6d11dc0bf8922850f6e6f648a3abd79c
I20260806 22:41:26.435791 29836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879: Bootstrap starting.
Aug 06 22:41:26 dist-test-slave-zr1t krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:26.439698 29836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:26.441100 29836 log.cc:824] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:26.444983 29836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879: No bootstrap required, opened a new log
I20260806 22:41:26.446812 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:26.423223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39302 (local address 127.23.100.190:45841)
0806 22:41:26.424174 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:26.424189 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:26.424244 (+ 55us) server_negotiation.cc:407] Connection header received
0806 22:41:26.425935 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:26.425975 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:26.426527 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:26.427041 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:26.428538 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:26.430546 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:26.431903 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:26.433037 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:26.433234 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:26.437651 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:26.437664 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:26.437686 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:26.437736 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:26.441686 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:26.442782 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:26.442791 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:26.442800 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:26.443046 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:26.443446 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:26.443451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:26.443453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:26.444198 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:26.444504 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:26.444959 (+ 455us) server_negotiation.cc:299] Negotiation successful
0806 22:41:26.445431 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":410,"threads_started":1}
I20260806 22:41:26.453872 29836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39877c5fca8d4448916bea31557a2879" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45841 } }
I20260806 22:41:26.454218 29836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:26.454291 29836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39877c5fca8d4448916bea31557a2879, State: Initialized, Role: FOLLOWER
I20260806 22:41:26.454806 29836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39877c5fca8d4448916bea31557a2879" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45841 } }
I20260806 22:41:26.455013 29836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:26.455106 29836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:26.455227 29836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:26.457816 29836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39877c5fca8d4448916bea31557a2879" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45841 } }
I20260806 22:41:26.458326 29836 leader_election.cc:302] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39877c5fca8d4448916bea31557a2879; no voters:
I20260806 22:41:26.458869 29836 leader_election.cc:288] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:26.459033 29846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:26.459728 29846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [term 1 LEADER]: Becoming Leader. State: Replica: 39877c5fca8d4448916bea31557a2879, State: Running, Role: LEADER
I20260806 22:41:26.460287 29846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39877c5fca8d4448916bea31557a2879" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45841 } }
I20260806 22:41:26.461026 29836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:26.463425 29850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39877c5fca8d4448916bea31557a2879. Latest consensus state: current_term: 1 leader_uuid: "39877c5fca8d4448916bea31557a2879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39877c5fca8d4448916bea31557a2879" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45841 } } }
I20260806 22:41:26.463398 29847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39877c5fca8d4448916bea31557a2879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39877c5fca8d4448916bea31557a2879" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45841 } } }
I20260806 22:41:26.463830 29847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:26.463828 29850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:26.465665 29853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:26.471215 29853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:26.480286 29853 catalog_manager.cc:1347] Generated new cluster ID: 0e8a7724c6d54077ad1777867bbba2db
I20260806 22:41:26.480425 29853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:26.498175 29853 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:26.500124 29853 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:26.513949 29853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879: Generated new TSK 0
I20260806 22:41:26.514842 29853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:41:26.580924 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45841
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:26.929692 29872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:26.930032 29872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:26.930212 29872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:26.937561 29872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:26.937723 29872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:26.937901 29872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:41:26.947974 29872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:26.949497 29872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:26.950922 29872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:26.964601 29879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:26.964891 29880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:26.964864 29883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:26.965974 29872 server_base.cc:1034] running on GCE node
I20260806 22:41:26.966571 29872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:26.967519 29872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:26.968742 29872 hybrid_clock.cc:648] HybridClock initialized: now 1786056086968671 us; error 55 us; skew 500 ppm
I20260806 22:41:26.969180 29872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:26 dist-test-slave-zr1t krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056086, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:26.976855 29872 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:41:26.978160 29872 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:26.978327 29872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:26.978669 29872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:26.981854 29872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "6d43c37de0a1466cb207142e5a6c222b"
format_stamp: "Formatted at 2026-08-06 22:41:26 on dist-test-slave-zr1t"
server_key: "09de00be01da39754dde87cfc20150a9"
server_key_iv: "4485d3ed47184ec37a8073789c8378b8"
server_key_version: "encryptionkey@0"
I20260806 22:41:26.982748 29872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "6d43c37de0a1466cb207142e5a6c222b"
format_stamp: "Formatted at 2026-08-06 22:41:26 on dist-test-slave-zr1t"
server_key: "09de00be01da39754dde87cfc20150a9"
server_key_iv: "4485d3ed47184ec37a8073789c8378b8"
server_key_version: "encryptionkey@0"
I20260806 22:41:26.987955 29872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 22:41:26.991665 29891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:26.992920 29872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:41:26.993104 29872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "6d43c37de0a1466cb207142e5a6c222b"
format_stamp: "Formatted at 2026-08-06 22:41:26 on dist-test-slave-zr1t"
server_key: "09de00be01da39754dde87cfc20150a9"
server_key_iv: "4485d3ed47184ec37a8073789c8378b8"
server_key_version: "encryptionkey@0"
I20260806 22:41:26.993335 29872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:27.014295 29872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:27.018999 29872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:27.019304 29872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:27.020306 29872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:27.021772 29872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:27.021910 29872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:27.022045 29872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:27.022141 29872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:27.058861 29872 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:43811
I20260806 22:41:27.058923 30056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:43811 every 8 connection(s)
I20260806 22:41:27.059772 29872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:41:27.067279 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 29872
I20260806 22:41:27.067554 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:41:27.068230 23954 external_mini_cluster.cc:1428] Setting key 23f42a942bf0135f67f4ade5e82b7a83
Aug 06 22:41:27 dist-test-slave-zr1t krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056086, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:27.081458 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:27.062455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:37101 (local address 127.23.100.190:45841)
0806 22:41:27.062977 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:27.062982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:27.063737 (+ 755us) server_negotiation.cc:407] Connection header received
0806 22:41:27.065635 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:27.065640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:27.065733 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:27.065854 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:27.067680 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:27.068520 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:27.070636 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:27.071063 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:27.071170 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:27.075565 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:27.075571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:27.075575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:27.075608 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:27.078327 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:27.079140 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:27.079145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:27.079147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:27.079274 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:27.079706 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:27.079710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:27.079712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:27.079984 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:27.080139 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:27.080847 (+ 708us) server_negotiation.cc:299] Negotiation successful
0806 22:41:27.081076 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":265,"threads_started":1}
I20260806 22:41:27.082471 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:27.062871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45841 (local address 127.23.100.129:37101)
0806 22:41:27.063567 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:27.063613 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:27.065327 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:27.066089 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:27.066098 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:27.066620 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:27.067477 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:27.067506 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:27.068685 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:27.068691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:27.070461 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:27.070477 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:27.071302 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:27.071311 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:27.071650 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:27.072462 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:27.072498 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:27.075314 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:27.078503 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:27.078512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:27.078530 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:27.078998 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:27.079384 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:27.079389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:27.079392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:27.079578 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:27.080119 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:27.080126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:27.080478 (+ 352us) client_negotiation.cc:769] Sending connection context
0806 22:41:27.080741 (+ 263us) client_negotiation.cc:240] Negotiation successful
0806 22:41:27.081305 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":282,"threads_started":1}
I20260806 22:41:27.084183 30059 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45841
I20260806 22:41:27.084524 30059 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:27.085335 30059 heartbeater.cc:511] Master 127.23.100.190:45841 requested a full tablet report, sending...
I20260806 22:41:27.087973 29783 ts_manager.cc:194] Registered new tserver with Master: 6d43c37de0a1466cb207142e5a6c222b (127.23.100.129:43811)
I20260806 22:41:27.090279 29783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:37101
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:41:27.124825 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45841
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:27.462905 30067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:27.463205 30067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:27.463306 30067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:27.471164 30067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:27.471280 30067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:27.471408 30067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:41:27.481475 30067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:27.482894 30067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:27.484143 30067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:27.497469 30076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:27.497495 30083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:27.497579 30077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:27.498637 30067 server_base.cc:1034] running on GCE node
I20260806 22:41:27.499246 30067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:27.500243 30067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:27.501576 30067 hybrid_clock.cc:648] HybridClock initialized: now 1786056087501445 us; error 68 us; skew 500 ppm
I20260806 22:41:27.501942 30067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:27 dist-test-slave-zr1t krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056087, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:27.505224 30067 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:41:27.506537 30067 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:27.506676 30067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:27.507038 30067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:27.510164 30067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "57ff07f9dd7343ce86955433c142ab23"
format_stamp: "Formatted at 2026-08-06 22:41:27 on dist-test-slave-zr1t"
server_key: "9139bb6852f495c1214e07f6e13a62aa"
server_key_iv: "3f0f46592def8072c9a86b6678ef21e1"
server_key_version: "encryptionkey@0"
I20260806 22:41:27.511051 30067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "57ff07f9dd7343ce86955433c142ab23"
format_stamp: "Formatted at 2026-08-06 22:41:27 on dist-test-slave-zr1t"
server_key: "9139bb6852f495c1214e07f6e13a62aa"
server_key_iv: "3f0f46592def8072c9a86b6678ef21e1"
server_key_version: "encryptionkey@0"
I20260806 22:41:27.516407 30067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:41:27.520056 30094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:27.521590 30067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:41:27.521777 30067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "57ff07f9dd7343ce86955433c142ab23"
format_stamp: "Formatted at 2026-08-06 22:41:27 on dist-test-slave-zr1t"
server_key: "9139bb6852f495c1214e07f6e13a62aa"
server_key_iv: "3f0f46592def8072c9a86b6678ef21e1"
server_key_version: "encryptionkey@0"
I20260806 22:41:27.522027 30067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:27.542995 30067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:27.547662 30067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:27.547957 30067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:27.549010 30067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:27.550570 30067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:27.550699 30067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:27.550840 30067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:27.550933 30067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:27.586086 30067 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:41005
I20260806 22:41:27.586148 30269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:41005 every 8 connection(s)
I20260806 22:41:27.587013 30067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:41:27.590150 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 30067
I20260806 22:41:27.590375 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:41:27.590883 23954 external_mini_cluster.cc:1428] Setting key bb13914278debfeb0b642ddccb104880
Aug 06 22:41:27 dist-test-slave-zr1t krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056087, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:27.609170 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:27.589621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:39713 (local address 127.23.100.190:45841)
0806 22:41:27.590286 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:27.590294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:27.591319 (+ 1025us) server_negotiation.cc:407] Connection header received
0806 22:41:27.593444 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:27.593450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:27.593584 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:27.593720 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:27.595280 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:27.596457 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:27.598437 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:27.599199 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:27.599354 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:27.603188 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:27.603193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:27.603197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:27.603230 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:27.605964 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:27.606758 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:27.606762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:27.606764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:27.606907 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:27.607368 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:27.607373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:27.607375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:27.607639 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:27.607806 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:27.608548 (+ 742us) server_negotiation.cc:299] Negotiation successful
0806 22:41:27.608796 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":369,"threads_started":1}
I20260806 22:41:27.610235 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:27.590213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45841 (local address 127.23.100.130:39713)
0806 22:41:27.591127 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:27.591176 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:27.593149 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:27.593901 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:27.593910 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:27.594481 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:27.595080 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:27.595099 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:27.596639 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:27.596643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:27.598265 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:27.598280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:27.599461 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:27.599467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:27.599683 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:27.600433 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:27.600455 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:27.602940 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:27.606134 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:27.606143 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:27.606162 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:27.606616 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:27.607029 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:27.607034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:27.607038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:27.607231 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:27.607777 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:27.607786 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:27.608150 (+ 364us) client_negotiation.cc:769] Sending connection context
0806 22:41:27.608438 (+ 288us) client_negotiation.cc:240] Negotiation successful
0806 22:41:27.608997 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":399,"threads_started":1}
I20260806 22:41:27.612051 30272 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45841
I20260806 22:41:27.612399 30272 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:27.613092 30272 heartbeater.cc:511] Master 127.23.100.190:45841 requested a full tablet report, sending...
I20260806 22:41:27.614835 29783 ts_manager.cc:194] Registered new tserver with Master: 57ff07f9dd7343ce86955433c142ab23 (127.23.100.130:41005)
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
I20260806 22:41:27.616211 29783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:39713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:41:27.646884 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45841
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:27.990477 30286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:27.990900 30286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:27.991091 30286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:27.998447 30286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:27.998600 30286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:27.998785 30286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:41:28.008706 30286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:28.010154 30286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:28.011507 30286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:28.025983 30292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:28.026099 30296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:28.026090 30299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:28.026971 30286 server_base.cc:1034] running on GCE node
I20260806 22:41:28.027587 30286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:28.028578 30286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:28.029804 30286 hybrid_clock.cc:648] HybridClock initialized: now 1786056088029753 us; error 46 us; skew 500 ppm
I20260806 22:41:28.030129 30286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:28 dist-test-slave-zr1t krb5kdc[29720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056088, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:28.033329 30286 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:41:28.034621 30286 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:28.034762 30286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:28.035089 30286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:28.038103 30286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "56ae87c9e74c49fc90f1d1219caf9c8b"
format_stamp: "Formatted at 2026-08-06 22:41:28 on dist-test-slave-zr1t"
server_key: "ae11cf0a006c85cf01529cfc1732c1c0"
server_key_iv: "f8fcf1e30caf3d285868f911c5a215fd"
server_key_version: "encryptionkey@0"
I20260806 22:41:28.038957 30286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "56ae87c9e74c49fc90f1d1219caf9c8b"
format_stamp: "Formatted at 2026-08-06 22:41:28 on dist-test-slave-zr1t"
server_key: "ae11cf0a006c85cf01529cfc1732c1c0"
server_key_iv: "f8fcf1e30caf3d285868f911c5a215fd"
server_key_version: "encryptionkey@0"
I20260806 22:41:28.044221 30286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:41:28.047829 30309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:28.049125 30286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:41:28.049306 30286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "56ae87c9e74c49fc90f1d1219caf9c8b"
format_stamp: "Formatted at 2026-08-06 22:41:28 on dist-test-slave-zr1t"
server_key: "ae11cf0a006c85cf01529cfc1732c1c0"
server_key_iv: "f8fcf1e30caf3d285868f911c5a215fd"
server_key_version: "encryptionkey@0"
I20260806 22:41:28.049600 30286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:28.070667 30286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:28.075070 30286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:28.075366 30286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:28.076388 30286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:28.077867 30286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:28.077986 30286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:28.078135 30286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:28.078225 30286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:28.094480 30059 heartbeater.cc:503] Master 127.23.100.190:45841 was elected leader, sending a full tablet report...
I20260806 22:41:28.113234 30286 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:33303
I20260806 22:41:28.113323 30485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:33303 every 8 connection(s)
I20260806 22:41:28.114169 30286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:41:28.121968 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 30286
I20260806 22:41:28.122187 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:41:28.122671 23954 external_mini_cluster.cc:1428] Setting key 843be5202a46afe52b78b6d63d18ebea
Aug 06 22:41:28 dist-test-slave-zr1t krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056088, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:28.134716 30490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.116634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:59131 (local address 127.23.100.190:45841)
0806 22:41:28.117149 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:28.117154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:28.117978 (+ 824us) server_negotiation.cc:407] Connection header received
0806 22:41:28.119858 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:28.119865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:28.119957 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:28.120079 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:28.121649 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.122559 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.124856 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.125247 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.125342 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.129160 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:28.129165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:28.129169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:28.129202 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:28.131989 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:28.132688 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:28.132692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:28.132694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:28.132806 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:28.133179 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:28.133182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:28.133184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:28.133429 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:28.133597 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:28.134268 (+ 671us) server_negotiation.cc:299] Negotiation successful
0806 22:41:28.134435 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":286,"threads_started":1}
I20260806 22:41:28.135646 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.117069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45841 (local address 127.23.100.131:59131)
0806 22:41:28.117809 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:28.117855 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:28.119566 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:28.120318 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:28.120327 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:28.120860 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:28.121441 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.121458 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.122730 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.122736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.124650 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.124667 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.125480 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.125488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.125815 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.126564 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:28.126588 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:28.128947 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:28.132122 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:28.132129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:28.132141 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:28.132578 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:28.132900 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:28.132903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:28.132906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:28.133080 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:28.133593 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:28.133599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:28.133934 (+ 335us) client_negotiation.cc:769] Sending connection context
0806 22:41:28.134162 (+ 228us) client_negotiation.cc:240] Negotiation successful
0806 22:41:28.134546 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":283,"threads_started":1}
I20260806 22:41:28.137159 30486 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45841
I20260806 22:41:28.137566 30486 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:28.138386 30486 heartbeater.cc:511] Master 127.23.100.190:45841 requested a full tablet report, sending...
I20260806 22:41:28.139808 29783 ts_manager.cc:194] Registered new tserver with Master: 56ae87c9e74c49fc90f1d1219caf9c8b (127.23.100.131:33303)
I20260806 22:41:28.140638 29783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:59131
I20260806 22:41:28.151091 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:41:28.169003 30490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.156044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.23.100.190:45841)
0806 22:41:28.156223 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:28.156228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:28.156771 (+ 543us) server_negotiation.cc:407] Connection header received
0806 22:41:28.156978 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:28.156986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:28.157060 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:28.157168 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:28.158744 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.159481 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.160780 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.161192 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.161345 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.163798 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:28.163803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:28.163806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:28.163840 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:28.166295 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:28.166975 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:28.166981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:28.166991 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:28.167120 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:28.167497 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:28.167504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:28.167507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:28.167838 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:28.168108 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:28.168415 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 22:41:28.168655 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 22:41:28.173113 29783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:28.180168 29783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:28.203874 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.192952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.23.100.131:33303)
0806 22:41:28.193942 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:28.193955 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:28.195016 (+ 1061us) server_negotiation.cc:407] Connection header received
0806 22:41:28.195237 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:28.195248 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:28.195524 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:28.195728 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:28.196443 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.198300 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.201371 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.202599 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.202834 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.203040 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:28.203318 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 22:41:28.203490 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":556,"threads_started":1}
I20260806 22:41:28.203992 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.192616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:33303 (local address 127.0.0.1:41488)
0806 22:41:28.194862 (+ 2246us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:28.194899 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:28.195043 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:28.195745 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:28.195754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:28.195782 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:28.196169 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.196185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.198536 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.198543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.201137 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.201154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.202830 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.202836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.203065 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.203119 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 22:41:28.203272 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 22:41:28.203483 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2021,"mutex_wait_us":6,"thread_start_us":2654,"threads_started":1}
I20260806 22:41:28.204769 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.193574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:41005 (local address 127.0.0.1:46384)
0806 22:41:28.195466 (+ 1892us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:28.195479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:28.195609 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:28.196266 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:28.196274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:28.196301 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:28.196790 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.196806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.199657 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.199667 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.201724 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.201736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.203886 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.203894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.204116 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.204151 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 22:41:28.204243 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 22:41:28.204377 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1798,"mutex_wait_us":27,"thread_start_us":334,"threads_started":1}
I20260806 22:41:28.204821 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.194034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.23.100.130:41005)
0806 22:41:28.194663 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:28.194675 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:28.195579 (+ 904us) server_negotiation.cc:407] Connection header received
0806 22:41:28.195824 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:28.195833 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:28.196049 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:28.196259 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:28.197279 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.199495 (+ 2216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.201944 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.203427 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.203588 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.203730 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:28.204323 (+ 593us) server_negotiation.cc:299] Negotiation successful
0806 22:41:28.204506 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":284,"threads_started":1}
I20260806 22:41:28.207158 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.196426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43811 (local address 127.0.0.1:54056)
0806 22:41:28.198512 (+ 2086us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:28.198530 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:28.198685 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:28.199241 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:28.199246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:28.199271 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:28.199853 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.199866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.201817 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.201824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.203965 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.203990 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.206399 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.206403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.206532 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.206551 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 22:41:28.206661 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 22:41:28.206789 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1938,"mutex_wait_us":1,"thread_start_us":1767,"threads_started":1}
I20260806 22:41:28.207196 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.196040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.23.100.129:43811)
0806 22:41:28.196911 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:28.196935 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:28.198639 (+ 1704us) server_negotiation.cc:407] Connection header received
0806 22:41:28.198866 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:28.198881 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:28.199063 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:28.199253 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:28.200058 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.201676 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.204155 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.206015 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.206155 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.206258 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:28.206707 (+ 449us) server_negotiation.cc:299] Negotiation successful
0806 22:41:28.206865 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":526,"threads_started":1}
I20260806 22:41:28.208410 30173 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1887429cd743ffa1f323733b389c77 (DEFAULT_TABLE table=test-table [id=4f00c2092d0d463b8cdcdfe130ef3da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:28.209873 30173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1887429cd743ffa1f323733b389c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:28.209559 30376 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1887429cd743ffa1f323733b389c77 (DEFAULT_TABLE table=test-table [id=4f00c2092d0d463b8cdcdfe130ef3da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:28.210937 30376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1887429cd743ffa1f323733b389c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:28.212641 29959 tablet_service.cc:1467] Processing CreateTablet for tablet 8f1887429cd743ffa1f323733b389c77 (DEFAULT_TABLE table=test-table [id=4f00c2092d0d463b8cdcdfe130ef3da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:28.214032 29959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f1887429cd743ffa1f323733b389c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:28.221903 30517 tablet_bootstrap.cc:492] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: Bootstrap starting.
I20260806 22:41:28.225816 30518 tablet_bootstrap.cc:492] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b: Bootstrap starting.
I20260806 22:41:28.225957 30517 tablet_bootstrap.cc:654] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:28.227444 30517 log.cc:824] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:28.228122 30519 tablet_bootstrap.cc:492] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: Bootstrap starting.
I20260806 22:41:28.230013 30518 tablet_bootstrap.cc:654] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:28.231333 30519 tablet_bootstrap.cc:654] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:28.231305 30518 log.cc:824] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:28.232589 30519 log.cc:824] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:28.232767 30517 tablet_bootstrap.cc:492] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: No bootstrap required, opened a new log
I20260806 22:41:28.233217 30517 ts_tablet_manager.cc:1397] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260806 22:41:28.236001 30518 tablet_bootstrap.cc:492] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b: No bootstrap required, opened a new log
I20260806 22:41:28.236249 30519 tablet_bootstrap.cc:492] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: No bootstrap required, opened a new log
I20260806 22:41:28.236326 30518 ts_tablet_manager.cc:1397] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260806 22:41:28.236511 30519 ts_tablet_manager.cc:1397] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260806 22:41:28.243863 30518 raft_consensus.cc:348] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 33303 } } peers { permanent_uuid: "6d43c37de0a1466cb207142e5a6c222b" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43811 } } peers { permanent_uuid: "57ff07f9dd7343ce86955433c142ab23" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41005 } }
I20260806 22:41:28.244315 30518 raft_consensus.cc:374] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:28.244086 30517 raft_consensus.cc:348] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 33303 } } peers { permanent_uuid: "6d43c37de0a1466cb207142e5a6c222b" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43811 } } peers { permanent_uuid: "57ff07f9dd7343ce86955433c142ab23" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41005 } }
I20260806 22:41:28.244422 30518 raft_consensus.cc:729] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ae87c9e74c49fc90f1d1219caf9c8b, State: Initialized, Role: FOLLOWER
I20260806 22:41:28.244534 30517 raft_consensus.cc:374] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:28.244647 30517 raft_consensus.cc:729] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ff07f9dd7343ce86955433c142ab23, State: Initialized, Role: FOLLOWER
I20260806 22:41:28.245092 30518 consensus_queue.cc:260] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 33303 } } peers { permanent_uuid: "6d43c37de0a1466cb207142e5a6c222b" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43811 } } peers { permanent_uuid: "57ff07f9dd7343ce86955433c142ab23" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41005 } }
I20260806 22:41:28.245319 30517 consensus_queue.cc:260] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 33303 } } peers { permanent_uuid: "6d43c37de0a1466cb207142e5a6c222b" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43811 } } peers { permanent_uuid: "57ff07f9dd7343ce86955433c142ab23" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41005 } }
I20260806 22:41:28.246270 30486 heartbeater.cc:503] Master 127.23.100.190:45841 was elected leader, sending a full tablet report...
I20260806 22:41:28.246767 30272 heartbeater.cc:503] Master 127.23.100.190:45841 was elected leader, sending a full tablet report...
I20260806 22:41:28.247027 30518 ts_tablet_manager.cc:1428] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260806 22:41:28.247169 30519 raft_consensus.cc:348] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 33303 } } peers { permanent_uuid: "6d43c37de0a1466cb207142e5a6c222b" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43811 } } peers { permanent_uuid: "57ff07f9dd7343ce86955433c142ab23" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41005 } }
I20260806 22:41:28.247571 30519 raft_consensus.cc:374] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:28.247715 30519 raft_consensus.cc:729] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d43c37de0a1466cb207142e5a6c222b, State: Initialized, Role: FOLLOWER
I20260806 22:41:28.248751 30517 ts_tablet_manager.cc:1428] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260806 22:41:28.248584 30519 consensus_queue.cc:260] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 33303 } } peers { permanent_uuid: "6d43c37de0a1466cb207142e5a6c222b" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43811 } } peers { permanent_uuid: "57ff07f9dd7343ce86955433c142ab23" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41005 } }
I20260806 22:41:28.250150 30519 ts_tablet_manager.cc:1428] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260806 22:41:28.270565 30527 raft_consensus.cc:482] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:28.270953 30527 raft_consensus.cc:504] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 33303 } } peers { permanent_uuid: "6d43c37de0a1466cb207142e5a6c222b" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43811 } } peers { permanent_uuid: "57ff07f9dd7343ce86955433c142ab23" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41005 } }
I20260806 22:41:28.272935 30527 leader_election.cc:288] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56ae87c9e74c49fc90f1d1219caf9c8b (127.23.100.131:33303), 6d43c37de0a1466cb207142e5a6c222b (127.23.100.129:43811)
I20260806 22:41:28.280018 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.273398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:50619 (local address 127.23.100.131:33303)
0806 22:41:28.273656 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:28.273661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:28.274375 (+ 714us) server_negotiation.cc:407] Connection header received
0806 22:41:28.274598 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:28.274604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:28.274682 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:28.274751 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:28.275509 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.276250 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.278132 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.279081 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.279255 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.279342 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:28.279632 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 22:41:28.279758 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 22:41:28.280335 30529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.273373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:33303 (local address 127.23.100.130:50619)
0806 22:41:28.274210 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:28.274230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:28.274374 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:28.274940 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:28.274944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:28.274974 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:28.275331 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.275339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.276378 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.276386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.277967 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.277979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.279318 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.279325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.279469 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.279492 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 22:41:28.279600 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 22:41:28.279737 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":17,"spinlock_wait_cycles":30464,"thread_start_us":471,"threads_started":1}
I20260806 22:41:28.280421 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.274559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:57253 (local address 127.23.100.129:43811)
0806 22:41:28.274776 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:28.274782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:28.274994 (+ 212us) server_negotiation.cc:407] Connection header received
0806 22:41:28.275233 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:28.275237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:28.275296 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:28.275408 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:28.275958 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.276662 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.278578 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.279490 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.279669 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.279743 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:28.280043 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 22:41:28.280168 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 22:41:28.280587 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.274282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43811 (local address 127.23.100.130:57253)
0806 22:41:28.274892 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:28.274906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:28.275068 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:28.275424 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:28.275428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:28.275451 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:28.275790 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.275799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.276812 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.276819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.278391 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:28.278403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.279705 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.279711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:28.279886 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.279909 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 22:41:28.280016 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 22:41:28.280148 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":288,"threads_started":1}
I20260806 22:41:28.281397 30404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1887429cd743ffa1f323733b389c77" candidate_uuid: "57ff07f9dd7343ce86955433c142ab23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" is_pre_election: true
I20260806 22:41:28.281603 29992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1887429cd743ffa1f323733b389c77" candidate_uuid: "57ff07f9dd7343ce86955433c142ab23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d43c37de0a1466cb207142e5a6c222b" is_pre_election: true
I20260806 22:41:28.282033 30404 raft_consensus.cc:2454] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ff07f9dd7343ce86955433c142ab23 in term 0.
I20260806 22:41:28.282168 29992 raft_consensus.cc:2454] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ff07f9dd7343ce86955433c142ab23 in term 0.
I20260806 22:41:28.282747 30105 leader_election.cc:302] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ae87c9e74c49fc90f1d1219caf9c8b, 57ff07f9dd7343ce86955433c142ab23; no voters:
I20260806 22:41:28.283152 30527 raft_consensus.cc:2781] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:28.283378 30527 raft_consensus.cc:482] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:28.283501 30527 raft_consensus.cc:3037] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:28.285877 30527 raft_consensus.cc:504] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 33303 } } peers { permanent_uuid: "6d43c37de0a1466cb207142e5a6c222b" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43811 } } peers { permanent_uuid: "57ff07f9dd7343ce86955433c142ab23" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41005 } }
I20260806 22:41:28.286662 30527 leader_election.cc:288] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [CANDIDATE]: Term 1 election: Requested vote from peers 56ae87c9e74c49fc90f1d1219caf9c8b (127.23.100.131:33303), 6d43c37de0a1466cb207142e5a6c222b (127.23.100.129:43811)
I20260806 22:41:28.287027 30404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1887429cd743ffa1f323733b389c77" candidate_uuid: "57ff07f9dd7343ce86955433c142ab23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b"
I20260806 22:41:28.287190 29992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f1887429cd743ffa1f323733b389c77" candidate_uuid: "57ff07f9dd7343ce86955433c142ab23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d43c37de0a1466cb207142e5a6c222b"
I20260806 22:41:28.287245 30404 raft_consensus.cc:3037] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:28.287421 29992 raft_consensus.cc:3037] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:28.289633 30404 raft_consensus.cc:2454] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ff07f9dd7343ce86955433c142ab23 in term 1.
I20260806 22:41:28.289834 29992 raft_consensus.cc:2454] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ff07f9dd7343ce86955433c142ab23 in term 1.
I20260806 22:41:28.290055 30105 leader_election.cc:302] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ae87c9e74c49fc90f1d1219caf9c8b, 57ff07f9dd7343ce86955433c142ab23; no voters:
I20260806 22:41:28.290349 30527 raft_consensus.cc:2781] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:28.290810 30527 raft_consensus.cc:686] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 1 LEADER]: Becoming Leader. State: Replica: 57ff07f9dd7343ce86955433c142ab23, State: Running, Role: LEADER
I20260806 22:41:28.291381 30527 consensus_queue.cc:237] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 33303 } } peers { permanent_uuid: "6d43c37de0a1466cb207142e5a6c222b" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43811 } } peers { permanent_uuid: "57ff07f9dd7343ce86955433c142ab23" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41005 } }
I20260806 22:41:28.296353 29783 catalog_manager.cc:5495] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57ff07f9dd7343ce86955433c142ab23 (127.23.100.130). New cstate: current_term: 1 leader_uuid: "57ff07f9dd7343ce86955433c142ab23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 33303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d43c37de0a1466cb207142e5a6c222b" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57ff07f9dd7343ce86955433c142ab23" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 41005 } health_report { overall_health: HEALTHY } } }
W20260806 22:41:28.312173 30060 tablet.cc:2369] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:41:28.329487 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.323848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.23.100.130:41005)
0806 22:41:28.324091 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:28.324098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:28.324210 (+ 112us) server_negotiation.cc:407] Connection header received
0806 22:41:28.324466 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:28.324470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:28.324562 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:28.324721 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:28.325134 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.325978 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.327352 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.327756 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.327890 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.328367 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:28.328885 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:28.329051 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:28.329111 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 22:41:28.329194 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 22:41:28.337798 30404 raft_consensus.cc:1261] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b [term 1 FOLLOWER]: Refusing update from remote peer 57ff07f9dd7343ce86955433c142ab23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:28.337800 29992 raft_consensus.cc:1261] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b [term 1 FOLLOWER]: Refusing update from remote peer 57ff07f9dd7343ce86955433c142ab23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 22:41:28.339028 30275 tablet.cc:2369] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:41:28.338824 30532 consensus_queue.cc:1035] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d43c37de0a1466cb207142e5a6c222b" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:28.339730 30527 consensus_queue.cc:1035] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ae87c9e74c49fc90f1d1219caf9c8b" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 33303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:28.351402 30535 mvcc.cc:205] Tried to move back new op lower bound from 7315685737821433856 to 7315685737646014464. Current Snapshot: MvccSnapshot[applied={T|T < 7315685737821433856}]
I20260806 22:41:28.364720 29780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:28.365059 29780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:28.369159 29780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39877c5fca8d4448916bea31557a2879: Sending DeleteTablet for 3 replicas of tablet 8f1887429cd743ffa1f323733b389c77
I20260806 22:41:28.370409 30376 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f1887429cd743ffa1f323733b389c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:28 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:41488
I20260806 22:41:28.370853 30173 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f1887429cd743ffa1f323733b389c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:28 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:46384
I20260806 22:41:28.371881 29959 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f1887429cd743ffa1f323733b389c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:28 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:54056
I20260806 22:41:28.373328 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 29872
I20260806 22:41:28.380929 30548 tablet_replica.cc:330] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: stopping tablet replica
I20260806 22:41:28.381565 30548 raft_consensus.cc:2229] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:28.382719 30547 tablet_replica.cc:330] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b: stopping tablet replica
I20260806 22:41:28.383060 30548 raft_consensus.cc:2258] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:28.383774 30547 raft_consensus.cc:2229] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:28.384284 30547 raft_consensus.cc:2258] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:28.386603 30548 ts_tablet_manager.cc:1905] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:28.394495 30548 ts_tablet_manager.cc:1918] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:28.395051 30549 tablet_replica.cc:330] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: stopping tablet replica
I20260806 22:41:28.395462 30549 raft_consensus.cc:2229] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:28.395918 30549 raft_consensus.cc:2258] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:28.396163 30547 ts_tablet_manager.cc:1905] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:28.398712 30549 ts_tablet_manager.cc:1905] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:28.401029 30548 log.cc:1196] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/8f1887429cd743ffa1f323733b389c77
I20260806 22:41:28.404263 30547 ts_tablet_manager.cc:1918] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:28.404464 30547 log.cc:1196] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/8f1887429cd743ffa1f323733b389c77
Aug 06 22:41:28 dist-test-slave-zr1t krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:41:28.405028 30547 ts_tablet_manager.cc:1939] T 8f1887429cd743ffa1f323733b389c77 P 56ae87c9e74c49fc90f1d1219caf9c8b: Deleting consensus metadata
I20260806 22:41:28.405030 30548 ts_tablet_manager.cc:1939] T 8f1887429cd743ffa1f323733b389c77 P 57ff07f9dd7343ce86955433c142ab23: Deleting consensus metadata
I20260806 22:41:28.406998 29764 catalog_manager.cc:4841] TS 56ae87c9e74c49fc90f1d1219caf9c8b (127.23.100.131:33303): tablet 8f1887429cd743ffa1f323733b389c77 (table test-table [id=4f00c2092d0d463b8cdcdfe130ef3da4]) successfully deleted
I20260806 22:41:28.408031 30549 ts_tablet_manager.cc:1918] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:28.408344 30549 log.cc:1196] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/8f1887429cd743ffa1f323733b389c77
I20260806 22:41:28.408380 29760 catalog_manager.cc:4841] TS 57ff07f9dd7343ce86955433c142ab23 (127.23.100.130:41005): tablet 8f1887429cd743ffa1f323733b389c77 (table test-table [id=4f00c2092d0d463b8cdcdfe130ef3da4]) successfully deleted
I20260806 22:41:28.408993 30549 ts_tablet_manager.cc:1939] T 8f1887429cd743ffa1f323733b389c77 P 6d43c37de0a1466cb207142e5a6c222b: Deleting consensus metadata
I20260806 22:41:28.411043 29764 catalog_manager.cc:4841] TS 6d43c37de0a1466cb207142e5a6c222b (127.23.100.129:43811): tablet 8f1887429cd743ffa1f323733b389c77 (table test-table [id=4f00c2092d0d463b8cdcdfe130ef3da4]) successfully deleted
I20260806 22:41:28.415434 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.377255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.23.100.129:43811)
0806 22:41:28.377487 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:28.377496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:28.382827 (+ 5331us) server_negotiation.cc:407] Connection header received
0806 22:41:28.391825 (+ 8998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:28.391836 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:28.391965 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:28.392058 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:28.396376 (+ 4318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.397545 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.399363 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.400015 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.400121 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.408165 (+ 8044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:28.408183 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:28.408202 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:28.408251 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:28.412399 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:28.413113 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:28.413124 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:28.413128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:28.413306 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:28.413747 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:28.413752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:28.413756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:28.414263 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:28.414617 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:28.414787 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 22:41:28.415029 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 22:41:28.416234 30051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:28.535753 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 29872
I20260806 22:41:28.564221 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 30067
Aug 06 22:41:28 dist-test-slave-zr1t krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:41:28.581449 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.565200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.23.100.130:41005)
0806 22:41:28.565420 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:28.565427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:28.565451 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:41:28.565547 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:28.565552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:28.565652 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:28.565763 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:28.567059 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.568118 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.569540 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.570089 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.570244 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.574709 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:28.574725 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:28.574741 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:28.574779 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:28.578389 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:28.579133 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:28.579143 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:28.579148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:28.579318 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:28.579729 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:28.579738 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:28.579742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:28.580096 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:28.580459 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:28.580760 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 22:41:28.581013 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 22:41:28.582535 30265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:28.699101 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 30067
I20260806 22:41:28.721253 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 30286
Aug 06 22:41:28 dist-test-slave-zr1t krb5kdc[29720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:41:28.737589 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:28.722068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.23.100.131:33303)
0806 22:41:28.722265 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:28.722272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:28.722346 (+ 74us) server_negotiation.cc:407] Connection header received
0806 22:41:28.722589 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:28.722593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:28.722659 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:28.722766 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:28.724104 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.724848 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.726396 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:28.726809 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:28.726950 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:28.731637 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:28.731655 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:28.731672 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:28.731712 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:28.734593 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:28.735353 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:28.735362 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:28.735364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:28.735516 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:28.735918 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:28.735923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:28.735925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:28.736295 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:28.736610 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:28.736945 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 22:41:28.737187 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 22:41:28.738312 30476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:28.830971 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 30286
I20260806 22:41:28.855862 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 29739
I20260806 22:41:28.857012 29829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:28.969692 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 29739
2026-08-06T22:41:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4601 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T22:41:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:29Z Disabled control of system clock
I20260806 22:41:29.057787 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:45901
--webserver_interface=127.23.100.190
--webserver_port=0
--builtin_ntp_servers=127.23.100.148:36727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:45901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 22:41:29.403934 30565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:29.404232 30565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:29.404377 30565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:29.411937 30565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:29.412060 30565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:29.412128 30565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:29.412214 30565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:29.412297 30565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:29.422024 30565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:45901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:45901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:29.423413 30565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:29.425325 30565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:29.437701 30572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:29.437654 30575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:29.437870 30573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:29.438974 30565 server_base.cc:1034] running on GCE node
I20260806 22:41:29.439884 30565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:29.441408 30565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:29.442710 30565 hybrid_clock.cc:648] HybridClock initialized: now 1786056089442650 us; error 38 us; skew 500 ppm
I20260806 22:41:29.443121 30565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:41:29.445446 30565 webserver.cc:460] Webserver started at http://127.23.100.190:39175/ using document root <none> and password file <none>
I20260806 22:41:29.446195 30565 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:29.446287 30565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:29.446573 30565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:29.449427 30565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "02fb8d4092a94a5b84699e34c465d397"
format_stamp: "Formatted at 2026-08-06 22:41:29 on dist-test-slave-zr1t"
server_key: "5fd76297d5e0afdb6b70d10772eb5390"
server_key_iv: "d5691665473af34df5051908dc51aa66"
server_key_version: "encryptionkey@0"
I20260806 22:41:29.450289 30565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "02fb8d4092a94a5b84699e34c465d397"
format_stamp: "Formatted at 2026-08-06 22:41:29 on dist-test-slave-zr1t"
server_key: "5fd76297d5e0afdb6b70d10772eb5390"
server_key_iv: "d5691665473af34df5051908dc51aa66"
server_key_version: "encryptionkey@0"
I20260806 22:41:29.455569 30565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:41:29.459105 30585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:29.460505 30565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 22:41:29.460690 30565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "02fb8d4092a94a5b84699e34c465d397"
format_stamp: "Formatted at 2026-08-06 22:41:29 on dist-test-slave-zr1t"
server_key: "5fd76297d5e0afdb6b70d10772eb5390"
server_key_iv: "d5691665473af34df5051908dc51aa66"
server_key_version: "encryptionkey@0"
I20260806 22:41:29.460927 30565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:29.466992 30565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:29.467793 30565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:29.468119 30565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:29.487504 30565 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:45901
I20260806 22:41:29.487537 30663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:45901 every 8 connection(s)
I20260806 22:41:29.488844 30565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:29.492290 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 30565
I20260806 22:41:29.492491 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:29.492688 30664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:29.493242 23954 external_mini_cluster.cc:1428] Setting key 75fd48bdffca85f1415afb2d58c179ba
I20260806 22:41:29.502686 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:29.495335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.23.100.190:45901)
0806 22:41:29.496467 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:29.496482 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:29.496525 (+ 43us) server_negotiation.cc:407] Connection header received
0806 22:41:29.498896 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:29.498919 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:29.499369 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:29.499726 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:29.500969 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:29.500976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:29.500988 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:29.501009 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:29.501080 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:29.501096 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:29.501274 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:29.501428 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 22:41:29.501739 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":489,"threads_started":1}
I20260806 22:41:29.506093 30664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397: Bootstrap starting.
I20260806 22:41:29.509140 30664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:29.510222 30664 log.cc:824] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:29.513118 30664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397: No bootstrap required, opened a new log
I20260806 22:41:29.520471 30664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fb8d4092a94a5b84699e34c465d397" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45901 } }
I20260806 22:41:29.520889 30664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:29.521014 30664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fb8d4092a94a5b84699e34c465d397, State: Initialized, Role: FOLLOWER
I20260806 22:41:29.521749 30664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fb8d4092a94a5b84699e34c465d397" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45901 } }
I20260806 22:41:29.522001 30664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:29.522158 30664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:29.522352 30664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:29.524541 30664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fb8d4092a94a5b84699e34c465d397" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45901 } }
I20260806 22:41:29.525055 30664 leader_election.cc:302] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02fb8d4092a94a5b84699e34c465d397; no voters:
I20260806 22:41:29.525753 30664 leader_election.cc:288] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:29.525931 30672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:29.526743 30672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [term 1 LEADER]: Becoming Leader. State: Replica: 02fb8d4092a94a5b84699e34c465d397, State: Running, Role: LEADER
I20260806 22:41:29.527344 30672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fb8d4092a94a5b84699e34c465d397" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45901 } }
I20260806 22:41:29.528060 30664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:29.530380 30674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02fb8d4092a94a5b84699e34c465d397. Latest consensus state: current_term: 1 leader_uuid: "02fb8d4092a94a5b84699e34c465d397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fb8d4092a94a5b84699e34c465d397" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45901 } } }
I20260806 22:41:29.530802 30674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:29.531723 30681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:29.531571 30673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02fb8d4092a94a5b84699e34c465d397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fb8d4092a94a5b84699e34c465d397" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45901 } } }
I20260806 22:41:29.532085 30673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:29.537918 30681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:29.546912 30681 catalog_manager.cc:1347] Generated new cluster ID: 271d7eda0cfb472aa6a4ba9c5aeda6b4
I20260806 22:41:29.547034 30681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:29.570214 30681 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:29.570676 30681 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:29.589198 30681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397: Generated new TSK 0
I20260806 22:41:29.590138 30681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:41:29.606989 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45901
--builtin_ntp_servers=127.23.100.148:36727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 22:41:29.958261 30699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:29.958629 30699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:29.958801 30699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:29.966591 30699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:29.966727 30699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:29.966909 30699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:41:29.976420 30699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:29.977864 30699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:29.979727 30699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:29.993669 30708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:29.993646 30712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:29.993583 30709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:29.994508 30699 server_base.cc:1034] running on GCE node
I20260806 22:41:29.995226 30699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:29.996284 30699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:29.997577 30699 hybrid_clock.cc:648] HybridClock initialized: now 1786056089997457 us; error 60 us; skew 500 ppm
I20260806 22:41:29.997975 30699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:41:30.000428 30699 webserver.cc:460] Webserver started at http://127.23.100.129:37519/ using document root <none> and password file <none>
I20260806 22:41:30.001159 30699 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:30.001255 30699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:30.001744 30699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:30.004740 30699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "6f63ab90d26e4537aeb1cdbd05712bb2"
format_stamp: "Formatted at 2026-08-06 22:41:30 on dist-test-slave-zr1t"
server_key: "6b71a795c9916eb5193a0f4425a1acff"
server_key_iv: "797a306b18836f68660b162f1919d080"
server_key_version: "encryptionkey@0"
I20260806 22:41:30.005666 30699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "6f63ab90d26e4537aeb1cdbd05712bb2"
format_stamp: "Formatted at 2026-08-06 22:41:30 on dist-test-slave-zr1t"
server_key: "6b71a795c9916eb5193a0f4425a1acff"
server_key_iv: "797a306b18836f68660b162f1919d080"
server_key_version: "encryptionkey@0"
I20260806 22:41:30.010887 30699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:41:30.014360 30721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:30.015679 30699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:30.015868 30699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "6f63ab90d26e4537aeb1cdbd05712bb2"
format_stamp: "Formatted at 2026-08-06 22:41:30 on dist-test-slave-zr1t"
server_key: "6b71a795c9916eb5193a0f4425a1acff"
server_key_iv: "797a306b18836f68660b162f1919d080"
server_key_version: "encryptionkey@0"
I20260806 22:41:30.016098 30699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:30.021905 30699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:30.022737 30699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:30.023033 30699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:30.023948 30699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:30.025413 30699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:30.025553 30699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:30.025699 30699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:30.025804 30699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:30.061029 30699 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:36605
I20260806 22:41:30.061126 30893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:36605 every 8 connection(s)
I20260806 22:41:30.062492 30699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:41:30.064195 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 30699
I20260806 22:41:30.064410 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:41:30.065279 23954 external_mini_cluster.cc:1428] Setting key 415b8dbfe3bb449f3310256e0f8b86d5
I20260806 22:41:30.068557 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45901
--builtin_ntp_servers=127.23.100.148:36727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 22:41:30.072734 30898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:30.065274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:46455 (local address 127.23.100.190:45901)
0806 22:41:30.065914 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:30.065919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:30.067342 (+ 1423us) server_negotiation.cc:407] Connection header received
0806 22:41:30.070133 (+ 2791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:30.070138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:30.070197 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:30.070367 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:30.071965 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:30.071970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:30.071973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:30.071992 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:30.072043 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:30.072058 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:30.072178 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:30.072410 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 22:41:30.072470 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":413,"threads_started":1}
I20260806 22:41:30.074050 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:30.066044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45901 (local address 127.23.100.129:46455)
0806 22:41:30.067171 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:30.067218 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:30.069716 (+ 2498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:30.070659 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:30.070673 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:30.070818 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:30.071570 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:41:30.071575 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:30.071626 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:41:30.071749 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:30.072167 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:30.072180 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:30.072191 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 22:41:30.072478 (+ 287us) client_negotiation.cc:240] Negotiation successful
0806 22:41:30.072748 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":365,"threads_started":1}
I20260806 22:41:30.075656 30894 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45901
I20260806 22:41:30.075970 30894 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:30.076587 30894 heartbeater.cc:511] Master 127.23.100.190:45901 requested a full tablet report, sending...
I20260806 22:41:30.078931 30613 ts_manager.cc:194] Registered new tserver with Master: 6f63ab90d26e4537aeb1cdbd05712bb2 (127.23.100.129:36605)
W20260806 22:41:30.412086 30901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:30.412395 30901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:30.412490 30901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:30.419920 30901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:30.420032 30901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:30.420142 30901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:41:30.429764 30901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:30.431010 30901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:30.432723 30901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:30.446524 30912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:30.446575 30917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:30.446475 30913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:30.447377 30901 server_base.cc:1034] running on GCE node
I20260806 22:41:30.447962 30901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:30.448869 30901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:30.450081 30901 hybrid_clock.cc:648] HybridClock initialized: now 1786056090450032 us; error 46 us; skew 500 ppm
I20260806 22:41:30.450376 30901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:41:30.452718 30901 webserver.cc:460] Webserver started at http://127.23.100.130:34589/ using document root <none> and password file <none>
I20260806 22:41:30.453397 30901 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:30.453490 30901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:30.453889 30901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:30.456947 30901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "bd00b6de381b430d948322452904ef16"
format_stamp: "Formatted at 2026-08-06 22:41:30 on dist-test-slave-zr1t"
server_key: "d221de1b6a03131c1807c66278716fb0"
server_key_iv: "8353c261b8f775efad3379bda448d1da"
server_key_version: "encryptionkey@0"
I20260806 22:41:30.457791 30901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "bd00b6de381b430d948322452904ef16"
format_stamp: "Formatted at 2026-08-06 22:41:30 on dist-test-slave-zr1t"
server_key: "d221de1b6a03131c1807c66278716fb0"
server_key_iv: "8353c261b8f775efad3379bda448d1da"
server_key_version: "encryptionkey@0"
I20260806 22:41:30.463073 30901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:41:30.466780 30929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:30.468343 30901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:30.468530 30901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "bd00b6de381b430d948322452904ef16"
format_stamp: "Formatted at 2026-08-06 22:41:30 on dist-test-slave-zr1t"
server_key: "d221de1b6a03131c1807c66278716fb0"
server_key_iv: "8353c261b8f775efad3379bda448d1da"
server_key_version: "encryptionkey@0"
I20260806 22:41:30.468782 30901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:30.474752 30901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:30.475610 30901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:30.475884 30901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:30.476825 30901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:30.478317 30901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:30.478425 30901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:30.478554 30901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:30.478654 30901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:30.516912 30901 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:33525
I20260806 22:41:30.516968 31081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:33525 every 8 connection(s)
I20260806 22:41:30.518306 30901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:41:30.526398 30898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:30.520897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:45607 (local address 127.23.100.190:45901)
0806 22:41:30.521111 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:30.521118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:30.522561 (+ 1443us) server_negotiation.cc:407] Connection header received
0806 22:41:30.524184 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:30.524188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:30.524243 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:30.524365 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:30.525712 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:30.525717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:30.525720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:30.525735 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:30.525773 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:30.525786 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:30.525882 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:30.526123 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 22:41:30.526180 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 22:41:30.526417 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 30901
I20260806 22:41:30.526582 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:41:30.527148 23954 external_mini_cluster.cc:1428] Setting key f80bf43140293936322dec48525b459a
I20260806 22:41:30.527611 31085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:30.521463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45901 (local address 127.23.100.130:45607)
0806 22:41:30.522392 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:30.522434 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:30.523943 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:30.524565 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:30.524572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:30.524655 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:30.525368 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:41:30.525371 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:30.525407 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:41:30.525495 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:30.525874 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:30.525882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:30.525887 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 22:41:30.526177 (+ 290us) client_negotiation.cc:240] Negotiation successful
0806 22:41:30.526430 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":266,"threads_started":1}
I20260806 22:41:30.529171 31082 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45901
I20260806 22:41:30.529475 31082 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:30.530040 31082 heartbeater.cc:511] Master 127.23.100.190:45901 requested a full tablet report, sending...
I20260806 22:41:30.530369 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45901
--builtin_ntp_servers=127.23.100.148:36727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 22:41:30.531548 30613 ts_manager.cc:194] Registered new tserver with Master: bd00b6de381b430d948322452904ef16 (127.23.100.130:33525)
W20260806 22:41:30.874374 31086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:30.874655 31086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:30.874826 31086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:30.882265 31086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:30.882375 31086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:30.882551 31086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:41:30.891614 31086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:30.892927 31086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:30.894721 31086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:30.908150 31095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:30.908200 31097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:30.908245 31100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:30.909109 31086 server_base.cc:1034] running on GCE node
I20260806 22:41:30.909845 31086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:30.910825 31086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:30.912078 31086 hybrid_clock.cc:648] HybridClock initialized: now 1786056090912001 us; error 56 us; skew 500 ppm
I20260806 22:41:30.912444 31086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:41:30.914821 31086 webserver.cc:460] Webserver started at http://127.23.100.131:35563/ using document root <none> and password file <none>
I20260806 22:41:30.915524 31086 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:30.915637 31086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:30.915984 31086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:30.918995 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "dc7425abe6384440802c1bfb7a6af805"
format_stamp: "Formatted at 2026-08-06 22:41:30 on dist-test-slave-zr1t"
server_key: "8942dc40aacc25db6cc8aa919025ef24"
server_key_iv: "7cb75a730e1f665040432b6a0a07b6cd"
server_key_version: "encryptionkey@0"
I20260806 22:41:30.919958 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "dc7425abe6384440802c1bfb7a6af805"
format_stamp: "Formatted at 2026-08-06 22:41:30 on dist-test-slave-zr1t"
server_key: "8942dc40aacc25db6cc8aa919025ef24"
server_key_iv: "7cb75a730e1f665040432b6a0a07b6cd"
server_key_version: "encryptionkey@0"
I20260806 22:41:30.925724 31086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 22:41:30.929697 31106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:30.931079 31086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:30.931265 31086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "dc7425abe6384440802c1bfb7a6af805"
format_stamp: "Formatted at 2026-08-06 22:41:30 on dist-test-slave-zr1t"
server_key: "8942dc40aacc25db6cc8aa919025ef24"
server_key_iv: "7cb75a730e1f665040432b6a0a07b6cd"
server_key_version: "encryptionkey@0"
I20260806 22:41:30.931510 31086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:30.937896 31086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:30.938796 31086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:30.939095 31086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:30.940110 31086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:30.941674 31086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:30.941790 31086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:30.941934 31086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:30.942026 31086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:30.976821 31086 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:46625
I20260806 22:41:30.976871 31274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:46625 every 8 connection(s)
I20260806 22:41:30.978268 31086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:41:30.980533 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 31086
I20260806 22:41:30.980729 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:41:30.981194 23954 external_mini_cluster.cc:1428] Setting key a368f66a80e60ff146e280bbba0fc50e
I20260806 22:41:30.987545 30898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:30.981721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:48469 (local address 127.23.100.190:45901)
0806 22:41:30.981915 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:30.981920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:30.983734 (+ 1814us) server_negotiation.cc:407] Connection header received
0806 22:41:30.985684 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:30.985689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:30.985749 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:30.985833 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:30.986994 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:30.986998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:30.987001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:30.987016 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:30.987048 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:30.987059 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:30.987144 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:30.987284 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 22:41:30.987332 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 22:41:30.988476 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:30.982348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45901 (local address 127.23.100.131:48469)
0806 22:41:30.983583 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:30.983632 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:30.985373 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:30.986012 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:30.986019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:30.986110 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:30.986713 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:41:30.986715 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:30.986752 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:41:30.986833 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:30.987121 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:30.987127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:30.987131 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 22:41:30.987331 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 22:41:30.987526 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":381,"threads_started":1}
I20260806 22:41:30.989909 31276 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45901
I20260806 22:41:30.990162 31276 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:30.990711 31276 heartbeater.cc:511] Master 127.23.100.190:45901 requested a full tablet report, sending...
I20260806 22:41:30.992024 30613 ts_manager.cc:194] Registered new tserver with Master: dc7425abe6384440802c1bfb7a6af805 (127.23.100.131:46625)
I20260806 22:41:30.996747 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:41:31.004076 30898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.001452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.23.100.190:45901)
0806 22:41:31.001718 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:31.001723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:31.002331 (+ 608us) server_negotiation.cc:407] Connection header received
0806 22:41:31.002472 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:31.002476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:31.002532 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:31.002623 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:31.003520 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:31.003524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:31.003527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:31.003542 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:31.003572 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:31.003584 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:31.003680 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:31.003806 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 22:41:31.003856 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260806 22:41:31.007977 30613 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:31.014840 30613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:31.034389 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.027828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:36605 (local address 127.0.0.1:41634)
0806 22:41:31.029005 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.029051 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:31.029185 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:31.030315 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:31.030325 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:31.030419 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:31.031469 (+ 1050us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:41:31.031475 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:31.031515 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:41:31.031629 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:31.033234 (+ 1605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:31.033296 (+ 62us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:31.033309 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 22:41:31.033511 (+ 202us) client_negotiation.cc:240] Negotiation successful
0806 22:41:31.033671 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":713,"threads_started":1}
I20260806 22:41:31.036634 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.028145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.23.100.129:36605)
0806 22:41:31.029689 (+ 1544us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:31.029699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:31.029733 (+ 34us) server_negotiation.cc:407] Connection header received
0806 22:41:31.029809 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:31.029817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:31.030067 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:31.031024 (+ 957us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:31.032853 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:31.032866 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:31.032885 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:31.032917 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:31.033041 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:31.033077 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:31.035674 (+ 2597us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:31.035910 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 22:41:31.036046 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1382,"thread_start_us":1204,"threads_started":1}
I20260806 22:41:31.041625 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.029998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:33525 (local address 127.0.0.1:55000)
0806 22:41:31.035229 (+ 5231us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.035266 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:31.035369 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:31.039218 (+ 3849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:31.039223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:31.039289 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:31.039877 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:41:31.039880 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:31.039899 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:41:31.039949 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:31.041126 (+ 1177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:31.041133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:31.041138 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 22:41:31.041220 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 22:41:31.041340 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5132,"threads_started":1}
I20260806 22:41:31.041345 30802 tablet_service.cc:1467] Processing CreateTablet for tablet 54b3d04ab06c444aace6ad0a78eb5bda (DEFAULT_TABLE table=test-table [id=44c92611c46641f6803ad73e044cb586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:31.042789 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.030351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.23.100.130:33525)
0806 22:41:31.038661 (+ 8310us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:31.038669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:31.038701 (+ 32us) server_negotiation.cc:407] Connection header received
0806 22:41:31.038772 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:31.038780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:31.039000 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:31.039598 (+ 598us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:31.040877 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:31.040885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:31.040900 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:31.040918 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:31.040970 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:31.040989 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:31.042165 (+ 1176us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:31.042336 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 22:41:31.042485 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8138,"threads_started":1}
I20260806 22:41:31.043152 30802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b3d04ab06c444aace6ad0a78eb5bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:31.049875 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.029500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.23.100.131:46625)
0806 22:41:31.043852 (+ 14352us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:31.043862 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:31.043898 (+ 36us) server_negotiation.cc:407] Connection header received
0806 22:41:31.043976 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:31.043983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:31.044182 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:31.044354 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:31.047838 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:31.047848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:31.047863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:31.047885 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:31.047955 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:31.047976 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:31.049196 (+ 1220us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:31.049320 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 22:41:31.049453 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14169,"thread_start_us":16403,"threads_started":1}
I20260806 22:41:31.050469 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.028810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:46625 (local address 127.0.0.1:52740)
0806 22:41:31.029698 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.029716 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:31.029821 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:31.045468 (+ 15647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:31.045473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:31.045499 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:31.046916 (+ 1417us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:41:31.046922 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:31.046944 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:41:31.047037 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:31.048505 (+ 1468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:31.048511 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:31.048516 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 22:41:31.048670 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 22:41:31.048730 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":764,"threads_started":1}
I20260806 22:41:31.050635 30989 tablet_service.cc:1467] Processing CreateTablet for tablet 54b3d04ab06c444aace6ad0a78eb5bda (DEFAULT_TABLE table=test-table [id=44c92611c46641f6803ad73e044cb586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:31.052663 30989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b3d04ab06c444aace6ad0a78eb5bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:31.056397 31175 tablet_service.cc:1467] Processing CreateTablet for tablet 54b3d04ab06c444aace6ad0a78eb5bda (DEFAULT_TABLE table=test-table [id=44c92611c46641f6803ad73e044cb586]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:31.057957 31175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b3d04ab06c444aace6ad0a78eb5bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:31.059788 31297 tablet_bootstrap.cc:492] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2: Bootstrap starting.
I20260806 22:41:31.064658 31297 tablet_bootstrap.cc:654] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:31.067593 31297 log.cc:824] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:31.077942 31301 tablet_bootstrap.cc:492] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: Bootstrap starting.
I20260806 22:41:31.079908 31297 tablet_bootstrap.cc:492] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2: No bootstrap required, opened a new log
I20260806 22:41:31.079999 31303 tablet_bootstrap.cc:492] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: Bootstrap starting.
I20260806 22:41:31.080210 31297 ts_tablet_manager.cc:1397] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.009s
I20260806 22:41:31.082093 30894 heartbeater.cc:503] Master 127.23.100.190:45901 was elected leader, sending a full tablet report...
I20260806 22:41:31.083148 31301 tablet_bootstrap.cc:654] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:31.083881 31303 tablet_bootstrap.cc:654] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:31.084708 31301 log.cc:824] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:31.085137 31303 log.cc:824] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:31.090626 31303 tablet_bootstrap.cc:492] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: No bootstrap required, opened a new log
I20260806 22:41:31.090632 31301 tablet_bootstrap.cc:492] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: No bootstrap required, opened a new log
I20260806 22:41:31.090939 31301 ts_tablet_manager.cc:1397] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260806 22:41:31.090936 31303 ts_tablet_manager.cc:1397] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260806 22:41:31.090842 31297 raft_consensus.cc:348] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 36605 } } peers { permanent_uuid: "dc7425abe6384440802c1bfb7a6af805" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46625 } } peers { permanent_uuid: "bd00b6de381b430d948322452904ef16" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 33525 } }
I20260806 22:41:31.091277 31297 raft_consensus.cc:374] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:31.091406 31297 raft_consensus.cc:729] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f63ab90d26e4537aeb1cdbd05712bb2, State: Initialized, Role: FOLLOWER
I20260806 22:41:31.092029 31297 consensus_queue.cc:260] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 36605 } } peers { permanent_uuid: "dc7425abe6384440802c1bfb7a6af805" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46625 } } peers { permanent_uuid: "bd00b6de381b430d948322452904ef16" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 33525 } }
I20260806 22:41:31.093909 31297 ts_tablet_manager.cc:1428] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2: Time spent starting tablet: real 0.014s user 0.005s sys 0.011s
I20260806 22:41:31.102046 31301 raft_consensus.cc:348] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 36605 } } peers { permanent_uuid: "dc7425abe6384440802c1bfb7a6af805" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46625 } } peers { permanent_uuid: "bd00b6de381b430d948322452904ef16" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 33525 } }
I20260806 22:41:31.102051 31303 raft_consensus.cc:348] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 36605 } } peers { permanent_uuid: "dc7425abe6384440802c1bfb7a6af805" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46625 } } peers { permanent_uuid: "bd00b6de381b430d948322452904ef16" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 33525 } }
I20260806 22:41:31.102574 31301 raft_consensus.cc:374] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:31.102578 31303 raft_consensus.cc:374] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:31.102687 31301 raft_consensus.cc:729] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd00b6de381b430d948322452904ef16, State: Initialized, Role: FOLLOWER
I20260806 22:41:31.102687 31303 raft_consensus.cc:729] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7425abe6384440802c1bfb7a6af805, State: Initialized, Role: FOLLOWER
I20260806 22:41:31.103364 31301 consensus_queue.cc:260] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 36605 } } peers { permanent_uuid: "dc7425abe6384440802c1bfb7a6af805" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46625 } } peers { permanent_uuid: "bd00b6de381b430d948322452904ef16" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 33525 } }
I20260806 22:41:31.103364 31303 consensus_queue.cc:260] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 36605 } } peers { permanent_uuid: "dc7425abe6384440802c1bfb7a6af805" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46625 } } peers { permanent_uuid: "bd00b6de381b430d948322452904ef16" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 33525 } }
I20260806 22:41:31.104516 31276 heartbeater.cc:503] Master 127.23.100.190:45901 was elected leader, sending a full tablet report...
I20260806 22:41:31.104965 31082 heartbeater.cc:503] Master 127.23.100.190:45901 was elected leader, sending a full tablet report...
I20260806 22:41:31.105356 31303 ts_tablet_manager.cc:1428] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 22:41:31.105589 31301 ts_tablet_manager.cc:1428] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260806 22:41:31.212119 31314 raft_consensus.cc:482] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:31.212428 31314 raft_consensus.cc:504] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 36605 } } peers { permanent_uuid: "dc7425abe6384440802c1bfb7a6af805" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46625 } } peers { permanent_uuid: "bd00b6de381b430d948322452904ef16" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 33525 } }
I20260806 22:41:31.214211 31314 leader_election.cc:288] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f63ab90d26e4537aeb1cdbd05712bb2 (127.23.100.129:36605), bd00b6de381b430d948322452904ef16 (127.23.100.130:33525)
I20260806 22:41:31.217144 31279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.214583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:36605 (local address 127.23.100.131:35947)
0806 22:41:31.214926 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.214949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:31.215059 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:31.215490 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:31.215495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:31.215519 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:31.216238 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:41:31.216241 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:31.216256 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:41:31.216301 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:31.216687 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:31.216690 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:31.216694 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 22:41:31.216789 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 22:41:31.216869 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260806 22:41:31.217145 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.214745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:35947 (local address 127.23.100.129:36605)
0806 22:41:31.214970 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:31.214979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:31.215010 (+ 31us) server_negotiation.cc:407] Connection header received
0806 22:41:31.215249 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:31.215255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:31.215324 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:31.215444 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:31.216469 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:31.216474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:31.216477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:31.216494 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:31.216553 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:31.216573 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:31.216706 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:31.216804 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 22:41:31.216874 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 22:41:31.217705 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.215010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:51883 (local address 127.23.100.130:33525)
0806 22:41:31.215207 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:31.215212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:31.215624 (+ 412us) server_negotiation.cc:407] Connection header received
0806 22:41:31.215809 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:31.215813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:31.215859 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:31.215964 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:31.216919 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:31.216925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:31.216929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:31.216947 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:31.216985 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:31.217002 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:31.217135 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:31.217315 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 22:41:31.217386 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 22:41:31.218227 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.214703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:33525 (local address 127.23.100.131:51883)
0806 22:41:31.215500 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.215519 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:31.215663 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:31.215992 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:31.215999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:31.216025 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:31.216685 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:41:31.216688 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:31.216704 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:41:31.216752 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:31.217132 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:31.217140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:31.217146 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 22:41:31.217845 (+ 699us) client_negotiation.cc:240] Negotiation successful
0806 22:41:31.217901 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":480,"threads_started":1}
I20260806 22:41:31.218375 30828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3d04ab06c444aace6ad0a78eb5bda" candidate_uuid: "dc7425abe6384440802c1bfb7a6af805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" is_pre_election: true
I20260806 22:41:31.219044 30828 raft_consensus.cc:2454] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc7425abe6384440802c1bfb7a6af805 in term 0.
I20260806 22:41:31.219267 31017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3d04ab06c444aace6ad0a78eb5bda" candidate_uuid: "dc7425abe6384440802c1bfb7a6af805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd00b6de381b430d948322452904ef16" is_pre_election: true
I20260806 22:41:31.219841 31017 raft_consensus.cc:2454] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc7425abe6384440802c1bfb7a6af805 in term 0.
I20260806 22:41:31.219825 31109 leader_election.cc:302] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f63ab90d26e4537aeb1cdbd05712bb2, dc7425abe6384440802c1bfb7a6af805; no voters:
I20260806 22:41:31.220286 31314 raft_consensus.cc:2781] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:31.220471 31314 raft_consensus.cc:482] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:31.220592 31314 raft_consensus.cc:3037] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:31.223080 31314 raft_consensus.cc:504] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 36605 } } peers { permanent_uuid: "dc7425abe6384440802c1bfb7a6af805" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46625 } } peers { permanent_uuid: "bd00b6de381b430d948322452904ef16" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 33525 } }
I20260806 22:41:31.223737 31314 leader_election.cc:288] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [CANDIDATE]: Term 1 election: Requested vote from peers 6f63ab90d26e4537aeb1cdbd05712bb2 (127.23.100.129:36605), bd00b6de381b430d948322452904ef16 (127.23.100.130:33525)
I20260806 22:41:31.224217 31017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3d04ab06c444aace6ad0a78eb5bda" candidate_uuid: "dc7425abe6384440802c1bfb7a6af805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd00b6de381b430d948322452904ef16"
I20260806 22:41:31.224220 30828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3d04ab06c444aace6ad0a78eb5bda" candidate_uuid: "dc7425abe6384440802c1bfb7a6af805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2"
I20260806 22:41:31.224437 31017 raft_consensus.cc:3037] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:31.224468 30828 raft_consensus.cc:3037] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:31.226823 31017 raft_consensus.cc:2454] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc7425abe6384440802c1bfb7a6af805 in term 1.
I20260806 22:41:31.227142 30828 raft_consensus.cc:2454] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc7425abe6384440802c1bfb7a6af805 in term 1.
I20260806 22:41:31.227331 31112 leader_election.cc:302] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd00b6de381b430d948322452904ef16, dc7425abe6384440802c1bfb7a6af805; no voters:
I20260806 22:41:31.227682 31314 raft_consensus.cc:2781] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:31.228219 31314 raft_consensus.cc:686] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 1 LEADER]: Becoming Leader. State: Replica: dc7425abe6384440802c1bfb7a6af805, State: Running, Role: LEADER
I20260806 22:41:31.228798 31314 consensus_queue.cc:237] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 36605 } } peers { permanent_uuid: "dc7425abe6384440802c1bfb7a6af805" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46625 } } peers { permanent_uuid: "bd00b6de381b430d948322452904ef16" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 33525 } }
W20260806 22:41:31.229913 31277 tablet.cc:2369] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:41:31.233726 30613 catalog_manager.cc:5495] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc7425abe6384440802c1bfb7a6af805 (127.23.100.131). New cstate: current_term: 1 leader_uuid: "dc7425abe6384440802c1bfb7a6af805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 36605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc7425abe6384440802c1bfb7a6af805" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd00b6de381b430d948322452904ef16" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 33525 } health_report { overall_health: UNKNOWN } } }
I20260806 22:41:31.249830 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.247531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.23.100.131:46625)
0806 22:41:31.247709 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:31.247716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:31.247804 (+ 88us) server_negotiation.cc:407] Connection header received
0806 22:41:31.248073 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:31.248078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:31.248156 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:31.248270 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:31.249191 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:31.249199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:31.249207 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:31.249223 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:31.249253 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:31.249265 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:31.249393 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:31.249502 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 22:41:31.249590 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 22:41:31.258103 30828 raft_consensus.cc:1261] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2 [term 1 FOLLOWER]: Refusing update from remote peer dc7425abe6384440802c1bfb7a6af805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:31.258103 31017 raft_consensus.cc:1261] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16 [term 1 FOLLOWER]: Refusing update from remote peer dc7425abe6384440802c1bfb7a6af805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:31.259332 31314 consensus_queue.cc:1035] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f63ab90d26e4537aeb1cdbd05712bb2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:31.259709 31317 consensus_queue.cc:1035] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd00b6de381b430d948322452904ef16" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 33525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 22:41:31.270263 31084 tablet.cc:2369] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:41:31.270538 31322 mvcc.cc:205] Tried to move back new op lower bound from 7315685749782589440 to 7315685749677789184. Current Snapshot: MvccSnapshot[applied={T|T < 7315685749782589440}]
I20260806 22:41:31.270951 31325 mvcc.cc:205] Tried to move back new op lower bound from 7315685749782589440 to 7315685749677789184. Current Snapshot: MvccSnapshot[applied={T|T < 7315685749782589440}]
I20260806 22:41:31.286170 30611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:31.286477 30611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:31.287169 31323 mvcc.cc:205] Tried to move back new op lower bound from 7315685749782589440 to 7315685749677789184. Current Snapshot: MvccSnapshot[applied={T|T < 7315685749782589440}]
I20260806 22:41:31.296113 30611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02fb8d4092a94a5b84699e34c465d397: Sending DeleteTablet for 3 replicas of tablet 54b3d04ab06c444aace6ad0a78eb5bda
I20260806 22:41:31.297327 30802 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b3d04ab06c444aace6ad0a78eb5bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:31 UTC) from {username='slave'} at 127.0.0.1:41634
I20260806 22:41:31.297498 30989 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b3d04ab06c444aace6ad0a78eb5bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:31 UTC) from {username='slave'} at 127.0.0.1:55000
I20260806 22:41:31.297479 31175 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b3d04ab06c444aace6ad0a78eb5bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:31 UTC) from {username='slave'} at 127.0.0.1:52740
I20260806 22:41:31.299546 31174 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b3d04ab06c444aace6ad0a78eb5bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:31 UTC) from {username='slave'} at 127.0.0.1:52740
I20260806 22:41:31.300766 30988 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b3d04ab06c444aace6ad0a78eb5bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:31 UTC) from {username='slave'} at 127.0.0.1:55000
I20260806 22:41:31.303764 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 30699
I20260806 22:41:31.308702 31335 tablet_replica.cc:330] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2: stopping tablet replica
I20260806 22:41:31.309316 31335 raft_consensus.cc:2229] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:31.310011 31335 raft_consensus.cc:2258] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:31.311105 31337 tablet_replica.cc:330] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: stopping tablet replica
I20260806 22:41:31.311646 31337 raft_consensus.cc:2229] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:31.312227 31337 raft_consensus.cc:2258] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:31.313066 31338 tablet_replica.cc:330] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: stopping tablet replica
I20260806 22:41:31.313511 31338 raft_consensus.cc:2229] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:31.314359 31338 raft_consensus.cc:2258] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:31.315285 31335 ts_tablet_manager.cc:1905] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:31.316609 31338 ts_tablet_manager.cc:1905] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:31.319758 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.305593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.23.100.129:36605)
0806 22:41:31.305831 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:31.305841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:31.316779 (+ 10938us) server_negotiation.cc:407] Connection header received
0806 22:41:31.317288 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:31.317294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:31.317395 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:31.317475 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:31.318830 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:31.318836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:31.318840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:31.318862 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:31.318903 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:31.318920 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:31.319054 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:31.319369 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 22:41:31.319441 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 22:41:31.320859 30886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:31.321368 31337 ts_tablet_manager.cc:1905] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:31.324791 31338 ts_tablet_manager.cc:1918] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:31.324983 31338 log.cc:1196] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/54b3d04ab06c444aace6ad0a78eb5bda
I20260806 22:41:31.325615 31338 ts_tablet_manager.cc:1939] T 54b3d04ab06c444aace6ad0a78eb5bda P dc7425abe6384440802c1bfb7a6af805: Deleting consensus metadata
I20260806 22:41:31.327638 30594 catalog_manager.cc:4841] TS dc7425abe6384440802c1bfb7a6af805 (127.23.100.131:46625): tablet 54b3d04ab06c444aace6ad0a78eb5bda (table test-table [id=44c92611c46641f6803ad73e044cb586]) successfully deleted
W20260806 22:41:31.328848 30594 catalog_manager.cc:4808] TS dc7425abe6384440802c1bfb7a6af805 (127.23.100.131:46625): delete failed for tablet 54b3d04ab06c444aace6ad0a78eb5bda because the tablet was not found. No further retry: Not found: Tablet not found: 54b3d04ab06c444aace6ad0a78eb5bda
I20260806 22:41:31.332839 31337 ts_tablet_manager.cc:1918] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:31.333091 31337 log.cc:1196] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/54b3d04ab06c444aace6ad0a78eb5bda
I20260806 22:41:31.333758 31337 ts_tablet_manager.cc:1939] T 54b3d04ab06c444aace6ad0a78eb5bda P bd00b6de381b430d948322452904ef16: Deleting consensus metadata
I20260806 22:41:31.335754 30594 catalog_manager.cc:4841] TS bd00b6de381b430d948322452904ef16 (127.23.100.130:33525): tablet 54b3d04ab06c444aace6ad0a78eb5bda (table test-table [id=44c92611c46641f6803ad73e044cb586]) successfully deleted
W20260806 22:41:31.336081 30594 catalog_manager.cc:4808] TS bd00b6de381b430d948322452904ef16 (127.23.100.130:33525): delete failed for tablet 54b3d04ab06c444aace6ad0a78eb5bda because the tablet was not found. No further retry: Not found: Tablet not found: 54b3d04ab06c444aace6ad0a78eb5bda
I20260806 22:41:31.425093 31335 ts_tablet_manager.cc:1918] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:31.425302 31335 log.cc:1196] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/54b3d04ab06c444aace6ad0a78eb5bda
I20260806 22:41:31.426015 31335 ts_tablet_manager.cc:1939] T 54b3d04ab06c444aace6ad0a78eb5bda P 6f63ab90d26e4537aeb1cdbd05712bb2: Deleting consensus metadata
I20260806 22:41:31.434417 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 30699
I20260806 22:41:31.465030 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 30901
W20260806 22:41:31.464946 30591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.100.129:36605 (error 108)
I20260806 22:41:31.467365 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.466805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:36605 (local address 127.0.0.1:41648)
0806 22:41:31.467021 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.467179 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 22:41:31.467649 30591 catalog_manager.cc:4584] TS 6f63ab90d26e4537aeb1cdbd05712bb2 (127.23.100.129:36605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54b3d04ab06c444aace6ad0a78eb5bda: Network error: Client connection negotiation failed: client connection to 127.23.100.129:36605: connect: Connection refused (error 111)
I20260806 22:41:31.469228 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.466373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55002 (local address 127.23.100.130:33525)
0806 22:41:31.466615 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:31.466623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:31.466644 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:41:31.466714 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:31.466720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:31.466836 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:31.466943 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:31.468541 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:31.468547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:31.468558 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:31.468577 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:31.468610 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:31.468623 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:31.468769 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:31.468863 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 22:41:31.468936 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 22:41:31.470111 31076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:31.521425 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.520819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:36605 (local address 127.0.0.1:41656)
0806 22:41:31.521093 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.521214 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260806 22:41:31.522737 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.522282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:36605 (local address 127.0.0.1:41668)
0806 22:41:31.522477 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.522579 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 22:41:31.560988 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 30901
I20260806 22:41:31.583528 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.583033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:36605 (local address 127.0.0.1:41684)
0806 22:41:31.583253 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.583363 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 22:41:31.584740 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.584310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:36605 (local address 127.0.0.1:41690)
0806 22:41:31.584510 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.584606 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 22:41:31.592975 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 31086
I20260806 22:41:31.595988 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.593778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.23.100.131:46625)
0806 22:41:31.593991 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:31.593997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:31.594021 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:41:31.594179 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:31.594185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:31.594250 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:31.594373 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:31.595280 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:31.595287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:31.595292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:31.595319 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:31.595359 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:31.595375 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:31.595532 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:31.595588 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 22:41:31.595655 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 22:41:31.596765 31267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:31.665607 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.664897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:36605 (local address 127.0.0.1:41700)
0806 22:41:31.665131 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.665247 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 22:41:31.667071 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.666588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:36605 (local address 127.0.0.1:41712)
0806 22:41:31.666871 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.666947 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260806 22:41:31.717212 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 31086
I20260806 22:41:31.745275 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 30565
I20260806 22:41:31.746628 30656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:31.858812 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:31.858163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:36605 (local address 127.0.0.1:41726)
0806 22:41:31.858410 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:31.858534 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 22:41:31.859062 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 30565
2026-08-06T22:41:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2890 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 22:41:31.909157 23954 mini_oidc.cc:152] Starting JWKS server
I20260806 22:41:31.912084 23954 webserver.cc:460] Webserver started at https://127.0.0.1:45453/ using document root <none> and password file <none>
I20260806 22:41:31.912362 23954 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 22:41:31.912999 23954 webserver.cc:460] Webserver started at http://127.0.0.1:35841/ using document root <none> and password file <none>
2026-08-06T22:41:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:31Z Disabled control of system clock
I20260806 22:41:31.951565 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33103
--webserver_interface=127.23.100.190
--webserver_port=0
--builtin_ntp_servers=127.23.100.148:36587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:33103
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45453/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/testchainca.cert with env {}
W20260806 22:41:32.286689 31357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:32.286985 31357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:32.287137 31357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:32.294417 31357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:32.294569 31357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:32.294668 31357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:32.294775 31357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:32.294886 31357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 22:41:32.294983 31357 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 22:41:32.295044 31357 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45453/jwks.json
I20260806 22:41:32.305370 31357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:33103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45453/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:32.306777 31357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:32.308094 31357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:32.320365 31364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:32.320511 31368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:32.320856 31372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:32.321429 31357 server_base.cc:1034] running on GCE node
I20260806 22:41:32.322227 31357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:32.323674 31357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:32.324893 31357 hybrid_clock.cc:648] HybridClock initialized: now 1786056092324853 us; error 37 us; skew 500 ppm
I20260806 22:41:32.325264 31357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:41:32.335136 31357 webserver.cc:460] Webserver started at http://127.23.100.190:37179/ using document root <none> and password file <none>
I20260806 22:41:32.335922 31357 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:32.336051 31357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:32.336375 31357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:32.339383 31357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "fc372f044f464e41a8f3ea734844c414"
format_stamp: "Formatted at 2026-08-06 22:41:32 on dist-test-slave-zr1t"
server_key: "a94bd7aadd50a42870e0c941b4283fad"
server_key_iv: "329d9c1863e8b23edf21b82d5944adc5"
server_key_version: "encryptionkey@0"
I20260806 22:41:32.340200 31357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "fc372f044f464e41a8f3ea734844c414"
format_stamp: "Formatted at 2026-08-06 22:41:32 on dist-test-slave-zr1t"
server_key: "a94bd7aadd50a42870e0c941b4283fad"
server_key_iv: "329d9c1863e8b23edf21b82d5944adc5"
server_key_version: "encryptionkey@0"
I20260806 22:41:32.345455 31357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 22:41:32.348901 31380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:32.350492 31357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:32.350735 31357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "fc372f044f464e41a8f3ea734844c414"
format_stamp: "Formatted at 2026-08-06 22:41:32 on dist-test-slave-zr1t"
server_key: "a94bd7aadd50a42870e0c941b4283fad"
server_key_iv: "329d9c1863e8b23edf21b82d5944adc5"
server_key_version: "encryptionkey@0"
I20260806 22:41:32.351019 31357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:32.375851 31357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:32.376837 31357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:32.377197 31357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:32.397854 31357 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:33103
I20260806 22:41:32.397897 31455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:33103 every 8 connection(s)
I20260806 22:41:32.399298 31357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:32.403120 31457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:32.405658 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 31357
I20260806 22:41:32.405886 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:32.406533 23954 external_mini_cluster.cc:1428] Setting key 8361fd80f77a8e025acae36b9e021587
I20260806 22:41:32.416776 31457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414: Bootstrap starting.
I20260806 22:41:32.420676 31457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:32.422122 31457 log.cc:824] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:32.422833 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:32.408593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.23.100.190:33103)
0806 22:41:32.409708 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:32.409722 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:32.409772 (+ 50us) server_negotiation.cc:407] Connection header received
0806 22:41:32.412175 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:32.412209 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:32.412857 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:32.413349 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:32.413964 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:32.415934 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:32.417356 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:32.418418 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:32.418644 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:32.419800 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:32.419811 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:32.419829 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:32.420548 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:32.420681 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:32.420711 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:32.420834 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:32.421088 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 22:41:32.421454 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":346,"threads_started":1}
I20260806 22:41:32.425504 31457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414: No bootstrap required, opened a new log
I20260806 22:41:32.427131 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:41:32.434453 31457 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f044f464e41a8f3ea734844c414" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33103 } }
I20260806 22:41:32.434825 31457 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:32.434911 31457 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc372f044f464e41a8f3ea734844c414, State: Initialized, Role: FOLLOWER
I20260806 22:41:32.435391 31457 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f044f464e41a8f3ea734844c414" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33103 } }
I20260806 22:41:32.435590 31457 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:32.435734 31457 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:32.435920 31457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:32.438388 31457 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f044f464e41a8f3ea734844c414" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33103 } }
I20260806 22:41:32.438830 31457 leader_election.cc:302] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc372f044f464e41a8f3ea734844c414; no voters:
I20260806 22:41:32.439378 31457 leader_election.cc:288] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:32.439514 31464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:32.440267 31464 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [term 1 LEADER]: Becoming Leader. State: Replica: fc372f044f464e41a8f3ea734844c414, State: Running, Role: LEADER
I20260806 22:41:32.440867 31464 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f044f464e41a8f3ea734844c414" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33103 } }
I20260806 22:41:32.441576 31457 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:32.443658 31466 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc372f044f464e41a8f3ea734844c414. Latest consensus state: current_term: 1 leader_uuid: "fc372f044f464e41a8f3ea734844c414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f044f464e41a8f3ea734844c414" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33103 } } }
I20260806 22:41:32.444104 31466 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:32.444778 31465 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc372f044f464e41a8f3ea734844c414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f044f464e41a8f3ea734844c414" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33103 } } }
I20260806 22:41:32.445065 31465 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:32.446552 31479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:32.451514 31479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:32.460117 31479 catalog_manager.cc:1347] Generated new cluster ID: 64bbae2b7d534df8bc46a25a3d587da8
I20260806 22:41:32.460248 31479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:32.468901 31479 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:32.470702 31479 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:32.485746 31479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc372f044f464e41a8f3ea734844c414: Generated new TSK 0
I20260806 22:41:32.486806 31479 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 22:41:32.528179 31502 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 22:41:32.528398 31502 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 22:41:32.529652 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:32.507055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.23.100.190:33103)
0806 22:41:32.507336 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:32.507375 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:32.507928 (+ 553us) server_negotiation.cc:407] Connection header received
0806 22:41:32.508275 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:32.508282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:32.508382 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:32.508548 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0806 22:41:32.509079 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:32.510261 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:32.511681 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:32.512375 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:32.512504 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:32.512996 (+ 492us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 22:41:32.528908 (+ 15912us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45453/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 22:41:32.529305 (+ 397us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56046: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45453/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":103}
W20260806 22:41:32.529774 31462 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56046: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45453/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 22:41:32.531026 31501 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:41:32.506837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33103 (local address 127.0.0.1:56046)
0806 22:41:32.507815 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:32.507831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:32.507987 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:32.508576 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:32.508580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:32.508590 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0806 22:41:32.508920 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:32.508928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:32.510399 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:32.510403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:32.511525 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:32.511537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:32.512545 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:32.512549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:32.512678 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:32.512737 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 22:41:32.529217 (+ 16480us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45453/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 22:41:32.529496 (+ 279us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.100.190:33103: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45453/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":630,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.100.190:33103: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45453/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 22:41:32.533236 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 31357
I20260806 22:41:32.534412 31445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:32.629611 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 31357
2026-08-06T22:41:32Z chronyd exiting
I20260806 22:41:32.686702 23954 test_util.cc:175] -----------------------------------------------
I20260806 22:41:32.686848 23954 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786056046488940-23954-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (782 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 22:41:32.690663 23954 mini_oidc.cc:152] Starting JWKS server
I20260806 22:41:32.692595 23954 webserver.cc:460] Webserver started at https://127.0.0.1:38125/ using document root <none> and password file <none>
I20260806 22:41:32.692783 23954 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 22:41:32.693390 23954 webserver.cc:460] Webserver started at http://127.0.0.1:43343/ using document root <none> and password file <none>
2026-08-06T22:41:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:32Z Disabled control of system clock
I20260806 22:41:32.730710 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:46783
--webserver_interface=127.23.100.190
--webserver_port=0
--builtin_ntp_servers=127.23.100.148:37117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:46783
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38125/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/testchainca.cert with env {}
W20260806 22:41:33.070384 31516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:33.070698 31516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:33.070854 31516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:33.078531 31516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:33.078653 31516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:33.078751 31516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:33.078840 31516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:33.078915 31516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 22:41:33.079027 31516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 22:41:33.079088 31516 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38125/jwks.json
I20260806 22:41:33.089475 31516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:37117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:46783
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:46783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38125/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:33.090888 31516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:33.092226 31516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:33.104125 31524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:33.104223 31527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:33.104202 31525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:33.105455 31516 server_base.cc:1034] running on GCE node
I20260806 22:41:33.106433 31516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:33.108108 31516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:33.109408 31516 hybrid_clock.cc:648] HybridClock initialized: now 1786056093109364 us; error 55 us; skew 500 ppm
I20260806 22:41:33.109853 31516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:41:33.112522 31516 webserver.cc:460] Webserver started at http://127.23.100.190:34583/ using document root <none> and password file <none>
I20260806 22:41:33.113646 31516 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:33.113786 31516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:33.114243 31516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:33.117266 31516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "115b18d0c1944182b450b9ddbc5d8038"
format_stamp: "Formatted at 2026-08-06 22:41:33 on dist-test-slave-zr1t"
server_key: "f41679dbd607594feef89c650eee6ecd"
server_key_iv: "51c2e86325e411b9df2c75e72047a0cd"
server_key_version: "encryptionkey@0"
I20260806 22:41:33.118204 31516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "115b18d0c1944182b450b9ddbc5d8038"
format_stamp: "Formatted at 2026-08-06 22:41:33 on dist-test-slave-zr1t"
server_key: "f41679dbd607594feef89c650eee6ecd"
server_key_iv: "51c2e86325e411b9df2c75e72047a0cd"
server_key_version: "encryptionkey@0"
I20260806 22:41:33.124025 31516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:41:33.128165 31540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:33.129766 31516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 22:41:33.129990 31516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "115b18d0c1944182b450b9ddbc5d8038"
format_stamp: "Formatted at 2026-08-06 22:41:33 on dist-test-slave-zr1t"
server_key: "f41679dbd607594feef89c650eee6ecd"
server_key_iv: "51c2e86325e411b9df2c75e72047a0cd"
server_key_version: "encryptionkey@0"
I20260806 22:41:33.130231 31516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:33.171437 31516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:33.172415 31516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:33.172765 31516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:33.192276 31516 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:46783
I20260806 22:41:33.192296 31612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:46783 every 8 connection(s)
I20260806 22:41:33.193854 31516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:33.195762 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 31516
I20260806 22:41:33.195977 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:33.196835 23954 external_mini_cluster.cc:1428] Setting key de3c53f1fc2d7365c4d2b64f24c444e7
I20260806 22:41:33.198807 31614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:33.212049 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:33.199114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.23.100.190:46783)
0806 22:41:33.200228 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:33.200239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:33.200281 (+ 42us) server_negotiation.cc:407] Connection header received
0806 22:41:33.201994 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:33.202016 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:33.202495 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:33.202838 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:33.203492 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:33.204874 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:33.206865 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:33.207916 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:33.208083 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:33.209507 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:33.209563 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:33.209591 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:33.210200 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:33.210285 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:33.210313 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:33.210425 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:33.210689 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 22:41:33.211075 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":438,"threads_started":1}
I20260806 22:41:33.213238 31614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038: Bootstrap starting.
I20260806 22:41:33.216077 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:41:33.216789 31614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:33.217892 31614 log.cc:824] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:33.220944 31614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038: No bootstrap required, opened a new log
I20260806 22:41:33.229104 31614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115b18d0c1944182b450b9ddbc5d8038" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46783 } }
I20260806 22:41:33.229496 31614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:33.229645 31614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 115b18d0c1944182b450b9ddbc5d8038, State: Initialized, Role: FOLLOWER
I20260806 22:41:33.230266 31614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115b18d0c1944182b450b9ddbc5d8038" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46783 } }
I20260806 22:41:33.230557 31614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:33.230710 31614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:33.230914 31614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:33.233619 31614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115b18d0c1944182b450b9ddbc5d8038" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46783 } }
I20260806 22:41:33.234193 31614 leader_election.cc:302] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 115b18d0c1944182b450b9ddbc5d8038; no voters:
I20260806 22:41:33.234817 31614 leader_election.cc:288] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:33.234966 31619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:33.236344 31619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [term 1 LEADER]: Becoming Leader. State: Replica: 115b18d0c1944182b450b9ddbc5d8038, State: Running, Role: LEADER
I20260806 22:41:33.236838 31619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115b18d0c1944182b450b9ddbc5d8038" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46783 } }
I20260806 22:41:33.237074 31614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:33.239759 31620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "115b18d0c1944182b450b9ddbc5d8038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115b18d0c1944182b450b9ddbc5d8038" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46783 } } }
I20260806 22:41:33.240187 31620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:33.240723 31622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 115b18d0c1944182b450b9ddbc5d8038. Latest consensus state: current_term: 1 leader_uuid: "115b18d0c1944182b450b9ddbc5d8038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115b18d0c1944182b450b9ddbc5d8038" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46783 } } }
I20260806 22:41:33.241160 31622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:33.243664 31633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:33.248718 31633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:33.258621 31633 catalog_manager.cc:1347] Generated new cluster ID: 1e061c09e2a84fdead0a1f8f19ecab1e
I20260806 22:41:33.258759 31633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:33.279858 31633 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:33.281741 31633 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:33.291530 31633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 115b18d0c1944182b450b9ddbc5d8038: Generated new TSK 0
I20260806 22:41:33.292376 31633 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 22:41:33.334764 31669 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 22:41:33.334901 31669 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 22:41:33.336143 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:33.307052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.23.100.190:46783)
0806 22:41:33.307325 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:33.307332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:33.307707 (+ 375us) server_negotiation.cc:407] Connection header received
0806 22:41:33.307971 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:33.307976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:33.308067 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:33.308212 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0806 22:41:33.308837 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:33.309778 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:33.311899 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:33.312423 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:33.312533 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:33.313227 (+ 694us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 22:41:33.335454 (+ 22227us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 22:41:33.335888 (+ 434us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":101}
W20260806 22:41:33.336300 31617 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 22:41:33.336463 31668 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:41:33.306841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:46783 (local address 127.0.0.1:51656)
0806 22:41:33.307576 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:33.307599 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:33.307764 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:33.308258 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:33.308264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:33.308285 (+ 21us) client_negotiation.cc:189] Negotiated authn=JWT
0806 22:41:33.308634 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:33.308646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:33.309938 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:33.309947 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:33.311700 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:33.311717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:33.312636 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:33.312644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:33.312862 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:33.312917 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 22:41:33.335735 (+ 22818us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 22:41:33.336022 (+ 287us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.100.190:46783: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":374,"threads_started":1}
I20260806 22:41:33.337924 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 31516
I20260806 22:41:33.339234 31607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:33.433601 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 31516
2026-08-06T22:41:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (807 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 22:41:33.498554 23954 mini_oidc.cc:152] Starting JWKS server
I20260806 22:41:33.500546 23954 webserver.cc:460] Webserver started at https://127.0.0.1:42555/ using document root <none> and password file <none>
I20260806 22:41:33.500763 23954 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 22:41:33.501418 23954 webserver.cc:460] Webserver started at http://127.0.0.1:38991/ using document root <none> and password file <none>
2026-08-06T22:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:33Z Disabled control of system clock
I20260806 22:41:33.540300 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:45849
--webserver_interface=127.23.100.190
--webserver_port=0
--builtin_ntp_servers=127.23.100.148:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:45849
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42555/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 22:41:33.890705 31676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:33.891076 31676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:33.891232 31676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:33.898550 31676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:33.898705 31676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:33.898793 31676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:33.898900 31676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:33.898985 31676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 22:41:33.899082 31676 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 22:41:33.899155 31676 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42555/jwks.json
I20260806 22:41:33.908856 31676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:45849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:45849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42555/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:33.910223 31676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:33.912051 31676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:33.924324 31685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:33.924448 31687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:33.924414 31690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:33.925231 31676 server_base.cc:1034] running on GCE node
I20260806 22:41:33.926188 31676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:33.927739 31676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:33.928994 31676 hybrid_clock.cc:648] HybridClock initialized: now 1786056093928956 us; error 71 us; skew 500 ppm
I20260806 22:41:33.929378 31676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:41:33.931931 31676 webserver.cc:460] Webserver started at http://127.23.100.190:36601/ using document root <none> and password file <none>
I20260806 22:41:33.932695 31676 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:33.932842 31676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:33.933197 31676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:33.936426 31676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "77f1f5dba9bb4b3093dda31f793468c9"
format_stamp: "Formatted at 2026-08-06 22:41:33 on dist-test-slave-zr1t"
server_key: "c7ef8965a295a1c0efd3a8b90b4ff167"
server_key_iv: "d37f1f2925c58632b61151b78f2341cc"
server_key_version: "encryptionkey@0"
I20260806 22:41:33.937290 31676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "77f1f5dba9bb4b3093dda31f793468c9"
format_stamp: "Formatted at 2026-08-06 22:41:33 on dist-test-slave-zr1t"
server_key: "c7ef8965a295a1c0efd3a8b90b4ff167"
server_key_iv: "d37f1f2925c58632b61151b78f2341cc"
server_key_version: "encryptionkey@0"
I20260806 22:41:33.942932 31676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 22:41:33.946750 31700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:33.948192 31676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:33.948446 31676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "77f1f5dba9bb4b3093dda31f793468c9"
format_stamp: "Formatted at 2026-08-06 22:41:33 on dist-test-slave-zr1t"
server_key: "c7ef8965a295a1c0efd3a8b90b4ff167"
server_key_iv: "d37f1f2925c58632b61151b78f2341cc"
server_key_version: "encryptionkey@0"
I20260806 22:41:33.948691 31676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:33.962195 31676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:33.963140 31676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:33.963488 31676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:33.984133 31676 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:45849
I20260806 22:41:33.984170 31771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:45849 every 8 connection(s)
I20260806 22:41:33.985616 31676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:33.986718 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 31676
I20260806 22:41:33.986953 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:33.987577 23954 external_mini_cluster.cc:1428] Setting key edc5a34f88bf8beac5f982932165db4d
I20260806 22:41:33.990198 31772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:34.003891 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:33.989658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.23.100.190:45849)
0806 22:41:33.990821 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:33.990837 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:33.990898 (+ 61us) server_negotiation.cc:407] Connection header received
0806 22:41:33.993134 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:33.993162 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:33.993823 (+ 661us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:33.994292 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:33.994852 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:33.996941 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:33.998529 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:33.999570 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:33.999765 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:34.001065 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:34.001077 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:34.001096 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:34.001927 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:34.002042 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:34.002073 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:34.002200 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:34.002575 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 22:41:34.002899 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":355,"threads_started":1}
I20260806 22:41:34.004698 31772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9: Bootstrap starting.
I20260806 22:41:34.008132 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:41:34.008400 31772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:34.009424 31772 log.cc:824] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:34.012549 31772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9: No bootstrap required, opened a new log
I20260806 22:41:34.020069 31772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f1f5dba9bb4b3093dda31f793468c9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45849 } }
I20260806 22:41:34.020475 31772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:34.020586 31772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f1f5dba9bb4b3093dda31f793468c9, State: Initialized, Role: FOLLOWER
I20260806 22:41:34.021260 31772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f1f5dba9bb4b3093dda31f793468c9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45849 } }
I20260806 22:41:34.021520 31772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:34.021718 31772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:34.021910 31772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:34.024151 31772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f1f5dba9bb4b3093dda31f793468c9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45849 } }
I20260806 22:41:34.024719 31772 leader_election.cc:302] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77f1f5dba9bb4b3093dda31f793468c9; no voters:
I20260806 22:41:34.025424 31772 leader_election.cc:288] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:34.025615 31783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:34.026402 31783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [term 1 LEADER]: Becoming Leader. State: Replica: 77f1f5dba9bb4b3093dda31f793468c9, State: Running, Role: LEADER
I20260806 22:41:34.026976 31783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f1f5dba9bb4b3093dda31f793468c9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45849 } }
I20260806 22:41:34.027671 31772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:34.030364 31786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77f1f5dba9bb4b3093dda31f793468c9. Latest consensus state: current_term: 1 leader_uuid: "77f1f5dba9bb4b3093dda31f793468c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f1f5dba9bb4b3093dda31f793468c9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45849 } } }
I20260806 22:41:34.030858 31786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:34.030206 31784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77f1f5dba9bb4b3093dda31f793468c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f1f5dba9bb4b3093dda31f793468c9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45849 } } }
I20260806 22:41:34.031790 31784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:34.033089 31791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:34.039860 31791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:34.050760 31791 catalog_manager.cc:1347] Generated new cluster ID: f21ada5d9ab14ba6bf45490c75775e32
I20260806 22:41:34.050887 31791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:34.096757 31791 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:34.098352 31791 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:34.109122 31791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77f1f5dba9bb4b3093dda31f793468c9: Generated new TSK 0
I20260806 22:41:34.110083 31791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:41:34.136240 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:34.127821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.23.100.190:45849)
0806 22:41:34.128100 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:34.128109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:34.128552 (+ 443us) server_negotiation.cc:407] Connection header received
0806 22:41:34.128838 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:34.128842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:34.128921 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:34.129104 (+ 183us) server_negotiation.cc:226] Negotiated authn=JWT
0806 22:41:34.129713 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:34.130452 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:34.131888 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:34.132384 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:34.132543 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:34.133103 (+ 560us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 22:41:34.135586 (+ 2483us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42555/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 22:41:34.135923 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42555/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 22:41:34.136374 31778 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42555/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 22:41:34.138501 31819 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:41:34.127721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45849 (local address 127.0.0.1:49772)
0806 22:41:34.128417 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:34.128446 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:34.128642 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:34.129162 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:34.129168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:34.129181 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0806 22:41:34.129489 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:34.129499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:34.130616 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:34.130622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:34.131727 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:34.131739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:34.132610 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:34.132620 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:34.132779 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:34.132819 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 22:41:34.135825 (+ 3006us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42555/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 22:41:34.138024 (+ 2199us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.100.190:45849: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42555/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":1851,"thread_start_us":395,"threads_started":1}
I20260806 22:41:34.139545 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 31676
I20260806 22:41:34.141031 31764 generic_service.cc:196] Checking for leaks (request via RPC)
E20260806 22:41:34.148413 31820 webserver.cc:533] Webserver: sslize failed:
E20260806 22:41:34.148566 31820 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 22:41:34.238430 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 31676
2026-08-06T22:41:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (800 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T22:41:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:34Z Disabled control of system clock
I20260806 22:41:34.339671 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:39237
--webserver_interface=127.23.100.190
--webserver_port=0
--builtin_ntp_servers=127.23.100.148:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:39237
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/testchainca.cert with env {}
W20260806 22:41:34.690796 31825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:34.691133 31825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:34.691224 31825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:34.698729 31825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:34.698853 31825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:34.698892 31825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:34.698944 31825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:34.698989 31825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 22:41:34.699054 31825 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 22:41:34.699095 31825 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 22:41:34.709651 31825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:39237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:39237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:34.711015 31825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:34.712337 31825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:34.727344 31831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:34.727389 31833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:34.727789 31825 server_base.cc:1034] running on GCE node
W20260806 22:41:34.727774 31836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:34.729033 31825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:34.730839 31825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:34.732128 31825 hybrid_clock.cc:648] HybridClock initialized: now 1786056094732087 us; error 45 us; skew 500 ppm
I20260806 22:41:34.732573 31825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:41:34.735435 31825 webserver.cc:460] Webserver started at http://127.23.100.190:45035/ using document root <none> and password file <none>
I20260806 22:41:34.736208 31825 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:34.736361 31825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:34.736706 31825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:34.739816 31825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "bf76a4f4ae8045bda3373f9396069885"
format_stamp: "Formatted at 2026-08-06 22:41:34 on dist-test-slave-zr1t"
server_key: "94df7a00d091aef893957c42fa6394b7"
server_key_iv: "668b64cecb63fe68f82ece0dd0e092d2"
server_key_version: "encryptionkey@0"
I20260806 22:41:34.740711 31825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "bf76a4f4ae8045bda3373f9396069885"
format_stamp: "Formatted at 2026-08-06 22:41:34 on dist-test-slave-zr1t"
server_key: "94df7a00d091aef893957c42fa6394b7"
server_key_iv: "668b64cecb63fe68f82ece0dd0e092d2"
server_key_version: "encryptionkey@0"
I20260806 22:41:34.746272 31825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 22:41:34.750085 31848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:34.751659 31825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 22:41:34.751899 31825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "bf76a4f4ae8045bda3373f9396069885"
format_stamp: "Formatted at 2026-08-06 22:41:34 on dist-test-slave-zr1t"
server_key: "94df7a00d091aef893957c42fa6394b7"
server_key_iv: "668b64cecb63fe68f82ece0dd0e092d2"
server_key_version: "encryptionkey@0"
I20260806 22:41:34.752161 31825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:34.769498 31825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:34.770524 31825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:34.770845 31825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:34.791015 31825 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:39237
I20260806 22:41:34.791055 31918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:39237 every 8 connection(s)
I20260806 22:41:34.792672 31825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:34.796208 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 31825
I20260806 22:41:34.796427 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:34.797219 31920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:34.797329 23954 external_mini_cluster.cc:1428] Setting key bef5502afabb84d2b9bf5668d049be9d
I20260806 22:41:34.811589 31920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885: Bootstrap starting.
I20260806 22:41:34.814762 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:34.800481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.23.100.190:39237)
0806 22:41:34.801987 (+ 1506us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:34.802000 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:34.802052 (+ 52us) server_negotiation.cc:407] Connection header received
0806 22:41:34.804622 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:34.804653 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:34.805273 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:34.805891 (+ 618us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:34.806542 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:34.808101 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:34.809757 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:34.810544 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:34.810772 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:34.812210 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:34.812219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:34.812234 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:41:34.812809 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:34.812902 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:41:34.812939 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:34.813113 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:34.813346 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 22:41:34.813733 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":564,"threads_started":1}
I20260806 22:41:34.815052 31920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:34.816390 31920 log.cc:824] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:34.819463 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:41:34.819945 31920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885: No bootstrap required, opened a new log
I20260806 22:41:34.828852 31920 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf76a4f4ae8045bda3373f9396069885" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39237 } }
I20260806 22:41:34.829339 31920 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:34.829468 31920 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf76a4f4ae8045bda3373f9396069885, State: Initialized, Role: FOLLOWER
I20260806 22:41:34.830137 31920 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf76a4f4ae8045bda3373f9396069885" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39237 } }
I20260806 22:41:34.830417 31920 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:34.830583 31920 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:34.830796 31920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:34.833609 31920 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf76a4f4ae8045bda3373f9396069885" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39237 } }
I20260806 22:41:34.834149 31920 leader_election.cc:302] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf76a4f4ae8045bda3373f9396069885; no voters:
I20260806 22:41:34.834736 31920 leader_election.cc:288] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:34.834921 31926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:34.835700 31926 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [term 1 LEADER]: Becoming Leader. State: Replica: bf76a4f4ae8045bda3373f9396069885, State: Running, Role: LEADER
I20260806 22:41:34.836335 31926 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf76a4f4ae8045bda3373f9396069885" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39237 } }
I20260806 22:41:34.837328 31920 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:34.839951 31931 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf76a4f4ae8045bda3373f9396069885. Latest consensus state: current_term: 1 leader_uuid: "bf76a4f4ae8045bda3373f9396069885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf76a4f4ae8045bda3373f9396069885" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39237 } } }
I20260806 22:41:34.839953 31927 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf76a4f4ae8045bda3373f9396069885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf76a4f4ae8045bda3373f9396069885" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39237 } } }
I20260806 22:41:34.840390 31931 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:34.840476 31927 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:34.842712 31937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:34.848101 31937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:34.858784 31937 catalog_manager.cc:1347] Generated new cluster ID: a31bad7d78414f1ba181da5d2e74d00d
I20260806 22:41:34.858933 31937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:34.867789 31937 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:34.869244 31937 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:34.888413 31937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf76a4f4ae8045bda3373f9396069885: Generated new TSK 0
I20260806 22:41:34.889405 31937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:41:35.356952 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:34.908947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.23.100.190:39237)
0806 22:41:34.909201 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:34.909209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:34.909822 (+ 613us) server_negotiation.cc:407] Connection header received
0806 22:41:34.910103 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:34.910107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:34.910193 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:34.910377 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0806 22:41:34.910941 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:34.911721 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:34.913911 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:34.914464 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:34.914591 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:34.915262 (+ 671us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 22:41:35.356285 (+441023us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 22:41:35.356682 (+ 397us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56558: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 22:41:35.357105 31924 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56558: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 22:41:35.357354 31960 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:41:34.908930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:39237 (local address 127.0.0.1:56558)
0806 22:41:34.909686 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:34.909709 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:34.909878 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:34.910396 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:34.910402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:34.910417 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0806 22:41:34.910742 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:34.910754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:34.911883 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:34.911889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:34.913711 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:34.913730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:34.914670 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:34.914677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:34.914887 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:34.914938 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 22:41:35.356617 (+441679us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 22:41:35.356923 (+ 306us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.100.190:39237: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":367,"threads_started":1}
I20260806 22:41:35.358472 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 31825
I20260806 22:41:35.359748 31914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:35.463354 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 31825
2026-08-06T22:41:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1209 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (464 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (439 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:41:36 dist-test-slave-zr1t krb5kdc[31979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:41:36 dist-test-slave-zr1t krb5kdc[31979](info): set up 2 sockets
Aug 06 22:41:36 dist-test-slave-zr1t krb5kdc[31979](info): commencing operation
krb5kdc: starting...
W20260806 22:41:37.695206 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.004s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:41:37 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:41:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:41:37.976853 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:37305
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:37305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:38.328562 31997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:38.328913 31997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:38.329017 31997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:38.336409 31997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:38.336534 31997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:38.336583 31997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:38.336645 31997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:38.336691 31997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:38.346768 31997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:37305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:37305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:38.348142 31997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:38.349390 31997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:38.361939 32003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:38.362007 32007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:38.362226 32004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:38.362926 31997 server_base.cc:1034] running on GCE node
I20260806 22:41:38.363824 31997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:38.365422 31997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:38.366719 31997 hybrid_clock.cc:648] HybridClock initialized: now 1786056098366667 us; error 41 us; skew 500 ppm
I20260806 22:41:38.367169 31997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:38 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056098, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:38.374079 31997 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:41:38.375982 31997 webserver.cc:460] Webserver started at http://127.23.100.190:37905/ using document root <none> and password file <none>
I20260806 22:41:38.376792 31997 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:38.376964 31997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:38.377334 31997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:38.380877 31997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "e609b353a46e4b85a8680b50e18c8dc2"
format_stamp: "Formatted at 2026-08-06 22:41:38 on dist-test-slave-zr1t"
server_key: "6be201592d97785be068cba43c56ebfc"
server_key_iv: "1c1b069cb7e792190156d06571861807"
server_key_version: "encryptionkey@0"
I20260806 22:41:38.381916 31997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "e609b353a46e4b85a8680b50e18c8dc2"
format_stamp: "Formatted at 2026-08-06 22:41:38 on dist-test-slave-zr1t"
server_key: "6be201592d97785be068cba43c56ebfc"
server_key_iv: "1c1b069cb7e792190156d06571861807"
server_key_version: "encryptionkey@0"
I20260806 22:41:38.387967 31997 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 22:41:38.392163 32018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:38.393896 31997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:38.394122 31997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "e609b353a46e4b85a8680b50e18c8dc2"
format_stamp: "Formatted at 2026-08-06 22:41:38 on dist-test-slave-zr1t"
server_key: "6be201592d97785be068cba43c56ebfc"
server_key_iv: "1c1b069cb7e792190156d06571861807"
server_key_version: "encryptionkey@0"
I20260806 22:41:38.394371 31997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:38.409137 31997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:38.414199 31997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:38.414585 31997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:38.436055 31997 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:37305
I20260806 22:41:38.436089 32099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:37305 every 8 connection(s)
I20260806 22:41:38.437664 31997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:38.441641 32100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:38.445772 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 31997
I20260806 22:41:38.446022 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:38.446785 23954 external_mini_cluster.cc:1428] Setting key 41c82b7307bd5271ca42e18e167cc1d6
I20260806 22:41:38.452946 32100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2: Bootstrap starting.
I20260806 22:41:38.456760 32100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:38.458088 32100 log.cc:824] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:38.461863 32100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2: No bootstrap required, opened a new log
Aug 06 22:41:38 dist-test-slave-zr1t krb5kdc[31979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:38.470381 32100 raft_consensus.cc:348] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e609b353a46e4b85a8680b50e18c8dc2" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37305 } }
I20260806 22:41:38.470816 32100 raft_consensus.cc:374] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:38.470947 32100 raft_consensus.cc:729] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e609b353a46e4b85a8680b50e18c8dc2, State: Initialized, Role: FOLLOWER
I20260806 22:41:38.471726 32100 consensus_queue.cc:260] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e609b353a46e4b85a8680b50e18c8dc2" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37305 } }
I20260806 22:41:38.472020 32100 raft_consensus.cc:388] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:38.472198 32100 raft_consensus.cc:482] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:38.472415 32100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:38.473007 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:38.449006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.23.100.190:37305)
0806 22:41:38.450009 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:38.450020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:38.450078 (+ 58us) server_negotiation.cc:407] Connection header received
0806 22:41:38.452054 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:38.452081 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:38.452492 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:38.452849 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:38.454793 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:38.456155 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:38.458146 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:38.459327 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:38.459531 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:38.464676 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:38.464686 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:38.464706 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:38.464749 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:38.468313 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:38.469143 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:38.469151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:38.469157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:38.469364 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:38.469832 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:38.469837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:38.469839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:38.470419 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:38.470765 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:38.471223 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 22:41:38.471704 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":489,"threads_started":1}
I20260806 22:41:38.475040 32100 raft_consensus.cc:504] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e609b353a46e4b85a8680b50e18c8dc2" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37305 } }
I20260806 22:41:38.475586 32100 leader_election.cc:302] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e609b353a46e4b85a8680b50e18c8dc2; no voters:
I20260806 22:41:38.476285 32100 leader_election.cc:288] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:38.476372 32106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:38.477201 32106 raft_consensus.cc:686] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [term 1 LEADER]: Becoming Leader. State: Replica: e609b353a46e4b85a8680b50e18c8dc2, State: Running, Role: LEADER
I20260806 22:41:38.477833 32106 consensus_queue.cc:237] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e609b353a46e4b85a8680b50e18c8dc2" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37305 } }
I20260806 22:41:38.478675 32100 sys_catalog.cc:564] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:38.480856 32107 sys_catalog.cc:455] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e609b353a46e4b85a8680b50e18c8dc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e609b353a46e4b85a8680b50e18c8dc2" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37305 } } }
I20260806 22:41:38.480860 32108 sys_catalog.cc:455] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e609b353a46e4b85a8680b50e18c8dc2. Latest consensus state: current_term: 1 leader_uuid: "e609b353a46e4b85a8680b50e18c8dc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e609b353a46e4b85a8680b50e18c8dc2" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37305 } } }
I20260806 22:41:38.481360 32107 sys_catalog.cc:458] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:38.481364 32108 sys_catalog.cc:458] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:38.482442 32113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:38.488288 32113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:38.498241 32113 catalog_manager.cc:1347] Generated new cluster ID: 0b64268b314d470cb07b65effb4d33e4
I20260806 22:41:38.498443 32113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:38.513270 32113 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:38.514768 32113 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:38.525920 32113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e609b353a46e4b85a8680b50e18c8dc2: Generated new TSK 0
I20260806 22:41:38.527144 32113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:41:38.593501 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37305
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:38.960532 32132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:38.960860 32132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:38.961036 32132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:38.968235 32132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:38.968395 32132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:38.968591 32132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:41:38.978789 32132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:38.980207 32132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:38.981699 32132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:38.994987 32138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:38.995056 32139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:38.995028 32143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:38.995842 32132 server_base.cc:1034] running on GCE node
I20260806 22:41:38.996608 32132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:38.997632 32132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:38.998870 32132 hybrid_clock.cc:648] HybridClock initialized: now 1786056098998814 us; error 50 us; skew 500 ppm
I20260806 22:41:38.999212 32132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:38 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056098, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:39.002774 32132 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:41:39.004294 32132 webserver.cc:460] Webserver started at http://127.23.100.129:44079/ using document root <none> and password file <none>
I20260806 22:41:39.004987 32132 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:39.005118 32132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:39.005460 32132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:39.008455 32132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "7ee22e22e98e4541942ebb3f79c9e0ba"
format_stamp: "Formatted at 2026-08-06 22:41:39 on dist-test-slave-zr1t"
server_key: "06a8277594db32f0c1721c7f9f879726"
server_key_iv: "61b598b4ec506da826b80be3691e1a4f"
server_key_version: "encryptionkey@0"
I20260806 22:41:39.009238 32132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "7ee22e22e98e4541942ebb3f79c9e0ba"
format_stamp: "Formatted at 2026-08-06 22:41:39 on dist-test-slave-zr1t"
server_key: "06a8277594db32f0c1721c7f9f879726"
server_key_iv: "61b598b4ec506da826b80be3691e1a4f"
server_key_version: "encryptionkey@0"
I20260806 22:41:39.014617 32132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 22:41:39.018415 32155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:39.019979 32132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:39.020164 32132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "7ee22e22e98e4541942ebb3f79c9e0ba"
format_stamp: "Formatted at 2026-08-06 22:41:39 on dist-test-slave-zr1t"
server_key: "06a8277594db32f0c1721c7f9f879726"
server_key_iv: "61b598b4ec506da826b80be3691e1a4f"
server_key_version: "encryptionkey@0"
I20260806 22:41:39.020395 32132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:39.036959 32132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:39.041651 32132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:39.041980 32132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:39.043213 32132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:39.047304 32132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:39.047420 32132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:39.047560 32132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:39.047668 32132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 22:41:39.086441 32132 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:33087
I20260806 22:41:39.086481 32321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:33087 every 8 connection(s)
I20260806 22:41:39.087832 32132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:41:39.092963 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 32132
I20260806 22:41:39.093187 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:41:39.094092 23954 external_mini_cluster.cc:1428] Setting key 2c820d5fbef118daeb583655b5adbd0c
Aug 06 22:41:39 dist-test-slave-zr1t krb5kdc[31979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056098, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:39.110850 32327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:39.090719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:45003 (local address 127.23.100.190:37305)
0806 22:41:39.091191 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:39.091196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:39.092288 (+ 1092us) server_negotiation.cc:407] Connection header received
0806 22:41:39.095024 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:39.095030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:39.095149 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:39.095276 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:39.097099 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:39.098003 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:39.099836 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:39.100291 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:39.100451 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:39.104651 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:39.104658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:39.104661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:39.104699 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:39.107570 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:39.108379 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:39.108388 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:39.108392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:39.108535 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:39.108995 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:39.109000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:39.109003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:39.109286 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:39.109451 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:39.110226 (+ 775us) server_negotiation.cc:299] Negotiation successful
0806 22:41:39.110461 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":252,"threads_started":1}
I20260806 22:41:39.111876 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:39.091376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37305 (local address 127.23.100.129:45003)
0806 22:41:39.092121 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:39.092166 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:39.094730 (+ 2564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:39.095548 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:39.095557 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:39.096078 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:39.096907 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:39.096934 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:39.098212 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:39.098217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:39.099668 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:39.099683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:39.100555 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:39.100565 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:39.100844 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:39.101690 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:39.101716 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:39.104354 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:39.107736 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:39.107746 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:39.107771 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:39.108240 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:39.108659 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:39.108664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:39.108668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:39.108874 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:39.109431 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:39.109439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:39.109854 (+ 415us) client_negotiation.cc:769] Sending connection context
0806 22:41:39.110135 (+ 281us) client_negotiation.cc:240] Negotiation successful
0806 22:41:39.110704 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":307,"threads_started":1}
I20260806 22:41:39.113745 32323 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37305
I20260806 22:41:39.114091 32323 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:39.115079 32323 heartbeater.cc:511] Master 127.23.100.190:37305 requested a full tablet report, sending...
I20260806 22:41:39.117751 32052 ts_manager.cc:194] Registered new tserver with Master: 7ee22e22e98e4541942ebb3f79c9e0ba (127.23.100.129:33087)
I20260806 22:41:39.119910 32052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:45003
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:41:39.157061 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37305
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:39.498965 32331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:39.499358 32331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:39.499565 32331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:39.507236 32331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:39.507359 32331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:39.507550 32331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:41:39.517705 32331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:39.519111 32331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:39.520514 32331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:39.534951 32341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:39.534974 32339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:39.535056 32338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:39.536109 32331 server_base.cc:1034] running on GCE node
I20260806 22:41:39.536732 32331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:39.537762 32331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:39.539026 32331 hybrid_clock.cc:648] HybridClock initialized: now 1786056099538956 us; error 61 us; skew 500 ppm
I20260806 22:41:39.539318 32331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:39 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056099, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:39.542798 32331 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:41:39.544368 32331 webserver.cc:460] Webserver started at http://127.23.100.130:34147/ using document root <none> and password file <none>
I20260806 22:41:39.545077 32331 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:39.545231 32331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:39.545610 32331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:39.548659 32331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "dd723a6643ee450d8576e212ca9e68dd"
format_stamp: "Formatted at 2026-08-06 22:41:39 on dist-test-slave-zr1t"
server_key: "933ad79333db4fd84b08776f87a3db49"
server_key_iv: "0661d2b1c16f2e697e24eb619dc13851"
server_key_version: "encryptionkey@0"
I20260806 22:41:39.549504 32331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "dd723a6643ee450d8576e212ca9e68dd"
format_stamp: "Formatted at 2026-08-06 22:41:39 on dist-test-slave-zr1t"
server_key: "933ad79333db4fd84b08776f87a3db49"
server_key_iv: "0661d2b1c16f2e697e24eb619dc13851"
server_key_version: "encryptionkey@0"
I20260806 22:41:39.555033 32331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 22:41:39.558820 32350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:39.560232 32331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 22:41:39.560474 32331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "dd723a6643ee450d8576e212ca9e68dd"
format_stamp: "Formatted at 2026-08-06 22:41:39 on dist-test-slave-zr1t"
server_key: "933ad79333db4fd84b08776f87a3db49"
server_key_iv: "0661d2b1c16f2e697e24eb619dc13851"
server_key_version: "encryptionkey@0"
I20260806 22:41:39.560712 32331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:39.593276 32331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:39.597771 32331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:39.598075 32331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:39.599186 32331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:39.600591 32331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:39.600664 32331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:39.600850 32331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:39.600904 32331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:39.637813 32331 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:42573
I20260806 22:41:39.637871 32519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:42573 every 8 connection(s)
I20260806 22:41:39.639163 32331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:41:39.644932 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 32331
I20260806 22:41:39.645119 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:41:39.645737 23954 external_mini_cluster.cc:1428] Setting key b910fdb919f165f261225d45ad89f163
Aug 06 22:41:39 dist-test-slave-zr1t krb5kdc[31979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056099, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:39.662257 32523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:39.641915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:50727 (local address 127.23.100.190:37305)
0806 22:41:39.642413 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:39.642418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:39.643322 (+ 904us) server_negotiation.cc:407] Connection header received
0806 22:41:39.645653 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:39.645658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:39.645731 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:39.645860 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:39.647801 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:39.648849 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:39.650606 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:39.651373 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:39.651535 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:39.655147 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:39.655157 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:39.655164 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:39.655214 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:39.658720 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:39.659761 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:39.659766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:39.659768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:39.659911 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:39.660452 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:39.660455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:39.660457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:39.660727 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:39.660845 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:39.661798 (+ 953us) server_negotiation.cc:299] Negotiation successful
0806 22:41:39.661984 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
I20260806 22:41:39.663897 32524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:39.642420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37305 (local address 127.23.100.130:50727)
0806 22:41:39.643150 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:39.643187 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:39.645254 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:39.646166 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:39.646178 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:39.646778 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:39.647606 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:39.647634 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:39.649048 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:39.649052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:39.650441 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:39.650456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:39.651618 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:39.651623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:39.651824 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:39.652553 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:39.652576 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:39.654901 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:39.658871 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:39.658880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:39.658901 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:39.659626 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:39.660034 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:39.660042 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:39.660049 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:39.660322 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:39.660857 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:39.660867 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:39.661346 (+ 479us) client_negotiation.cc:769] Sending connection context
0806 22:41:39.661712 (+ 366us) client_negotiation.cc:240] Negotiation successful
0806 22:41:39.662335 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":298,"threads_started":1}
I20260806 22:41:39.665764 32520 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37305
I20260806 22:41:39.666115 32520 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:39.666879 32520 heartbeater.cc:511] Master 127.23.100.190:37305 requested a full tablet report, sending...
I20260806 22:41:39.668475 32052 ts_manager.cc:194] Registered new tserver with Master: dd723a6643ee450d8576e212ca9e68dd (127.23.100.130:42573)
I20260806 22:41:39.669831 32052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:50727
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:41:39.707162 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37305
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:40.055708 32528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:40.056020 32528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:40.056191 32528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:40.063669 32528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:40.063787 32528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:40.063968 32528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:41:40.074024 32528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:40.075325 32528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:40.076639 32528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:40.089998 32539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:40.090021 32540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:40.090965 32543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:40.092181 32528 server_base.cc:1034] running on GCE node
I20260806 22:41:40.092868 32528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:40.093920 32528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:40.095170 32528 hybrid_clock.cc:648] HybridClock initialized: now 1786056100095104 us; error 65 us; skew 500 ppm
I20260806 22:41:40.095512 32528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:40.098904 32528 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:41:40.100409 32528 webserver.cc:460] Webserver started at http://127.23.100.131:41677/ using document root <none> and password file <none>
I20260806 22:41:40.101100 32528 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:40.101203 32528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:40.101466 32528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:40.104534 32528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "7fe61eda013c4fc3990f53e83ae32247"
format_stamp: "Formatted at 2026-08-06 22:41:40 on dist-test-slave-zr1t"
server_key: "c7531a2028792972de0f887d8090eb13"
server_key_iv: "a966077319802409f829fe1a1470a425"
server_key_version: "encryptionkey@0"
I20260806 22:41:40.105317 32528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "7fe61eda013c4fc3990f53e83ae32247"
format_stamp: "Formatted at 2026-08-06 22:41:40 on dist-test-slave-zr1t"
server_key: "c7531a2028792972de0f887d8090eb13"
server_key_iv: "a966077319802409f829fe1a1470a425"
server_key_version: "encryptionkey@0"
I20260806 22:41:40.110771 32528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 22:41:40.114437 32552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:40.115792 32528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:40.115979 32528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "7fe61eda013c4fc3990f53e83ae32247"
format_stamp: "Formatted at 2026-08-06 22:41:40 on dist-test-slave-zr1t"
server_key: "c7531a2028792972de0f887d8090eb13"
server_key_iv: "a966077319802409f829fe1a1470a425"
server_key_version: "encryptionkey@0"
I20260806 22:41:40.116207 32528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:40.124037 32323 heartbeater.cc:503] Master 127.23.100.190:37305 was elected leader, sending a full tablet report...
I20260806 22:41:40.137703 32528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:40.142364 32528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:40.142683 32528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:40.143802 32528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:40.145226 32528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:40.145306 32528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:40.145457 32528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:40.145551 32528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:40.182502 32528 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:43469
I20260806 22:41:40.182511 32722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:43469 every 8 connection(s)
I20260806 22:41:40.184054 32528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:41:40.184564 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 32528
I20260806 22:41:40.184754 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:41:40.185439 23954 external_mini_cluster.cc:1428] Setting key ed79300a02530358f425a257aabac139
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:41:40.206775 32727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:40.186977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:41347 (local address 127.23.100.190:37305)
0806 22:41:40.187560 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:40.187568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:40.189728 (+ 2160us) server_negotiation.cc:407] Connection header received
0806 22:41:40.191323 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:40.191329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:40.191480 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:40.191605 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:40.193234 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.194304 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.195929 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.196426 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.196549 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:40.200666 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:40.200672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:40.200675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:40.200709 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:40.203608 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.204355 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.204362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.204366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.204502 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.204945 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.204951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.204954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.205231 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:40.205398 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:40.206148 (+ 750us) server_negotiation.cc:299] Negotiation successful
0806 22:41:40.206384 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":306,"threads_started":1}
I20260806 22:41:40.207732 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:40.187811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37305 (local address 127.23.100.131:41347)
0806 22:41:40.188892 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:40.188953 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:40.190984 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:40.191876 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:40.191885 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:40.192482 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:40.193044 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:40.193061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.194485 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.194489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:40.195763 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:40.195778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.196680 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.196686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:40.196901 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:40.197705 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:40.197735 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:40.200441 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:40.203753 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:40.203763 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:40.203788 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:40.204225 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:40.204614 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:40.204619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:40.204623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:40.204832 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:40.205364 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:40.205372 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:40.205798 (+ 426us) client_negotiation.cc:769] Sending connection context
0806 22:41:40.206073 (+ 275us) client_negotiation.cc:240] Negotiation successful
0806 22:41:40.206614 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":479,"threads_started":1}
I20260806 22:41:40.209261 32723 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37305
I20260806 22:41:40.209645 32723 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:40.210422 32723 heartbeater.cc:511] Master 127.23.100.190:37305 requested a full tablet report, sending...
I20260806 22:41:40.211887 32052 ts_manager.cc:194] Registered new tserver with Master: 7fe61eda013c4fc3990f53e83ae32247 (127.23.100.131:43469)
I20260806 22:41:40.212909 32052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:41347
I20260806 22:41:40.214524 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/test-admin.keytab.
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:40.249051 23954 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 22:41:40.254571 32734 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:41:40.251888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.23.100.129:33087)
0806 22:41:40.252550 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:40.252558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:40.252585 (+ 27us) server_negotiation.cc:407] Connection header received
0806 22:41:40.252647 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:40.252654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:40.252857 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:40.253003 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:40.254130 (+ 1127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32848: BlockingRecv error: recv got EOF from 127.0.0.1:32848 (error 108)
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":341,"threads_started":1}
W20260806 22:41:40.254733 23954 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/krb5cc' of type FILE
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:40.256791 23954 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:40.258425 23954 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:41:40.274534 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:40.260291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.23.100.129:33087)
0806 22:41:40.260483 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:40.260489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:40.260601 (+ 112us) server_negotiation.cc:407] Connection header received
0806 22:41:40.260796 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:40.260808 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:40.260868 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:40.260974 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:40.262133 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.263113 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.264378 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.264974 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.265109 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:40.268952 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:40.268966 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:40.268980 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:40.269026 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:40.271931 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.272636 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.272644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.272646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.272781 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.273107 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.273110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.273112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.273499 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:40.273807 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:40.274075 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 22:41:40.274278 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 22:41:40.280778 32734 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:41:40.278790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.23.100.129:33087)
0806 22:41:40.278985 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:40.278993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:40.279344 (+ 351us) server_negotiation.cc:407] Connection header received
0806 22:41:40.279559 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:40.279563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:40.279638 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:40.279739 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:40.280550 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32866: BlockingRecv error: recv got EOF from 127.0.0.1:32866 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 22:41:40.281066 23954 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/krb5cc' of type FILE
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:40.282806 23954 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:40.284238 23954 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:41:40.298843 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:40.285644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.23.100.129:33087)
0806 22:41:40.285874 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:40.285880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:40.286083 (+ 203us) server_negotiation.cc:407] Connection header received
0806 22:41:40.286206 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:40.286210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:40.286270 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:40.286349 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:40.287284 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.287973 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.289402 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.289885 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.289995 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:40.293690 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:40.293695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:40.293698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:40.293731 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:40.296359 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.297014 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.297018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.297021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.297139 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.297485 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.297488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.297490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.297801 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:40.298024 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:40.298343 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 22:41:40.298512 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260806 22:41:40.303915 32734 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:41:40.301598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.23.100.129:33087)
0806 22:41:40.301886 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:40.301893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:40.302169 (+ 276us) server_negotiation.cc:407] Connection header received
0806 22:41:40.302488 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:40.302494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:40.302591 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:40.302709 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:40.303614 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32872: BlockingRecv error: recv got EOF from 127.0.0.1:32872 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260806 22:41:40.304260 23954 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786056046488940-23954-0/krb5kdc/krb5cc' of type FILE
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:40.306020 23954 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:40.307466 23954 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:41:40.323968 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:40.308860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.23.100.129:33087)
0806 22:41:40.309097 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:40.309106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:40.309456 (+ 350us) server_negotiation.cc:407] Connection header received
0806 22:41:40.309712 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:40.309718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:40.309797 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:40.309908 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:40.310937 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.311747 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.313128 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.313711 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.313865 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:40.317862 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:40.317869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:40.317872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:40.317909 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:40.320919 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.321856 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.321863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.321867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.322033 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.322467 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.322471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.322473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.322756 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:40.323108 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:40.323517 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 22:41:40.323702 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 22:41:40.325769 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 32132
I20260806 22:41:40.339641 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:40.326797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.23.100.129:33087)
0806 22:41:40.327056 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:40.327062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:40.327310 (+ 248us) server_negotiation.cc:407] Connection header received
0806 22:41:40.327700 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:40.327706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:40.327783 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:40.327894 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:40.329004 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.329941 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.331223 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.331660 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.331800 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:40.334337 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:40.334342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:40.334345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:40.334382 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:40.336905 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.337669 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.337677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.337680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.337836 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.338223 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.338229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.338232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.338500 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:40.338784 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:40.339091 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 22:41:40.339316 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260806 22:41:40.340327 32315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:40.426226 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 32132
I20260806 22:41:40.455456 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 32331
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:41:40.472894 32751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:40.456563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52610 (local address 127.23.100.130:42573)
0806 22:41:40.457085 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:40.457092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:40.457118 (+ 26us) server_negotiation.cc:407] Connection header received
0806 22:41:40.457188 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:40.457195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:40.457404 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:40.457618 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:40.458669 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.459731 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.461014 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.461670 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.461815 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:40.466186 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:40.466199 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:40.466214 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:40.466253 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:40.470025 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.470789 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.470798 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.470800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.470962 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.471304 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.471307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.471309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.471683 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:40.471916 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:40.472234 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 22:41:40.472536 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
I20260806 22:41:40.474344 32510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:40.558609 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 32331
I20260806 22:41:40.582672 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 32528
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[31979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:41:40.599119 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:40.583798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.23.100.131:43469)
0806 22:41:40.584444 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:40.584477 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:40.584507 (+ 30us) server_negotiation.cc:407] Connection header received
0806 22:41:40.584567 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:40.584573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:40.584791 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:40.584926 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:40.586150 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.587172 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.588472 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:40.589122 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:40.589220 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:40.593657 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:40.593670 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:40.593684 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:40.593719 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:40.596420 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.597104 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.597110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.597113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.597231 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:40.597645 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:40.597652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:40.597657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:40.598022 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:40.598312 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:40.598610 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 22:41:40.598817 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":309,"threads_started":1}
I20260806 22:41:40.600507 32717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:40.683619 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 32528
I20260806 22:41:40.707870 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 31997
I20260806 22:41:40.709048 32095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:40.814402 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 31997
2026-08-06T22:41:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4453 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[32760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[32760](info): set up 2 sockets
Aug 06 22:41:40 dist-test-slave-zr1t krb5kdc[32760](info): commencing operation
krb5kdc: starting...
W20260806 22:41:42.141112 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:41:42 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:41:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.190.keytab.
Entry for principal oryx/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.190.keytab.
I20260806 22:41:42.420748 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:45503
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.190.keytab
--principal=oryx/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:42.764472 310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:42.764854 310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:42.765034 310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:42.772620 310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:42.772778 310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:42.772899 310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:42.773002 310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:42.773083 310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:42.783161 310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:45503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.190.keytab
--principal=oryx/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:42.784578 310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:42.786078 310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:42.799544 317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:42.799558 316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:42.800432 320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:42.801473 310 server_base.cc:1034] running on GCE node
I20260806 22:41:42.802417 310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:42.803929 310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:42.805166 310 hybrid_clock.cc:648] HybridClock initialized: now 1786056102805109 us; error 54 us; skew 500 ppm
I20260806 22:41:42.805576 310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:42 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056102, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:42.808883 310 init.cc:377] Logged in from keytab as oryx/127.23.100.190@KRBTEST.COM (short username oryx)
I20260806 22:41:42.810516 310 webserver.cc:460] Webserver started at http://127.23.100.190:46833/ using document root <none> and password file <none>
I20260806 22:41:42.811213 310 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:42.811362 310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:42.811674 310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:42.814725 310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "9a9e131374e743a4803b662ca72cf6dd"
format_stamp: "Formatted at 2026-08-06 22:41:42 on dist-test-slave-zr1t"
server_key: "943fd245428e11b18c8c578f5237e500"
server_key_iv: "93d8de19874525bc1344d81a4c59edf0"
server_key_version: "encryptionkey@0"
I20260806 22:41:42.815495 310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "9a9e131374e743a4803b662ca72cf6dd"
format_stamp: "Formatted at 2026-08-06 22:41:42 on dist-test-slave-zr1t"
server_key: "943fd245428e11b18c8c578f5237e500"
server_key_iv: "93d8de19874525bc1344d81a4c59edf0"
server_key_version: "encryptionkey@0"
I20260806 22:41:42.820793 310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:41:42.824337 332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:42.825747 310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 22:41:42.825932 310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "9a9e131374e743a4803b662ca72cf6dd"
format_stamp: "Formatted at 2026-08-06 22:41:42 on dist-test-slave-zr1t"
server_key: "943fd245428e11b18c8c578f5237e500"
server_key_iv: "93d8de19874525bc1344d81a4c59edf0"
server_key_version: "encryptionkey@0"
I20260806 22:41:42.826136 310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:42.845745 310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:42.849896 310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:42.850178 310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:42.869319 310 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:45503
I20260806 22:41:42.869410 402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:45503 every 8 connection(s)
I20260806 22:41:42.870771 310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:42.874900 403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:42.878154 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 310
I20260806 22:41:42.878389 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:42.879114 23954 external_mini_cluster.cc:1428] Setting key be15f86f68a43b9ba6a67da5781dcf2a
I20260806 22:41:42.889782 403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd: Bootstrap starting.
I20260806 22:41:42.893744 403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:42.894821 403 log.cc:824] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd: Log is configured to *not* fsync() on all Append() calls
Aug 06 22:41:42 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:42.898372 403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd: No bootstrap required, opened a new log
I20260806 22:41:42.904964 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:42.881416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.23.100.190:45503)
0806 22:41:42.882337 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:42.882355 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:42.882411 (+ 56us) server_negotiation.cc:407] Connection header received
0806 22:41:42.884944 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:42.884980 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:42.885621 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:42.886152 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:42.887707 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:42.889957 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:42.891242 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:42.892471 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:42.892668 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:42.897306 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:42.897314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:42.897329 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:42.897379 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:42.900690 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:42.901669 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:42.901675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:42.901682 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:42.901846 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:42.902300 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:42.902303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:42.902305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:42.902843 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:42.903098 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:42.903542 (+ 444us) server_negotiation.cc:299] Negotiation successful
0806 22:41:42.903860 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":351,"threads_started":1}
I20260806 22:41:42.908376 403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } }
I20260806 22:41:42.908769 403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:42.908933 403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9e131374e743a4803b662ca72cf6dd, State: Initialized, Role: FOLLOWER
I20260806 22:41:42.909610 403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } }
I20260806 22:41:42.909871 403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:42.910055 403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:42.910243 403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:42.912717 403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } }
I20260806 22:41:42.913259 403 leader_election.cc:302] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a9e131374e743a4803b662ca72cf6dd; no voters:
I20260806 22:41:42.914001 403 leader_election.cc:288] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:42.914121 411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:42.915143 411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 1 LEADER]: Becoming Leader. State: Replica: 9a9e131374e743a4803b662ca72cf6dd, State: Running, Role: LEADER
I20260806 22:41:42.915755 411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } }
I20260806 22:41:42.916478 403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:42.919341 414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a9e131374e743a4803b662ca72cf6dd. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } }
I20260806 22:41:42.919749 414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:42.919343 412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } }
I20260806 22:41:42.920639 412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:42.922327 426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:42.927861 426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:42.937345 426 catalog_manager.cc:1347] Generated new cluster ID: 9743b2c273644d7899b4f95bc4699be9
I20260806 22:41:42.937474 426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:42.945964 426 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:42.948015 426 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:42.963968 426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd: Generated new TSK 0
I20260806 22:41:42.965166 426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab.
Entry for principal oryx/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab.
I20260806 22:41:43.029062 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45503
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab
--principal=oryx/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:43.378158 443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:43.378451 443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:43.378548 443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:43.385733 443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:43.385840 443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:43.385958 443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:41:43.395596 443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab
--principal=oryx/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:43.396862 443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:43.398144 443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:43.412278 455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:43.412354 456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:43.412675 459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:43.413621 443 server_base.cc:1034] running on GCE node
I20260806 22:41:43.414309 443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:43.415257 443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:43.416543 443 hybrid_clock.cc:648] HybridClock initialized: now 1786056103416477 us; error 43 us; skew 500 ppm
I20260806 22:41:43.416879 443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:43 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056103, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:43.424252 443 init.cc:377] Logged in from keytab as oryx/127.23.100.129@KRBTEST.COM (short username oryx)
I20260806 22:41:43.425892 443 webserver.cc:460] Webserver started at http://127.23.100.129:45113/ using document root <none> and password file <none>
I20260806 22:41:43.426600 443 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:43.426756 443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:43.427083 443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:43.430469 443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "cd37baa871ee4ebbb82552241b0069b2"
format_stamp: "Formatted at 2026-08-06 22:41:43 on dist-test-slave-zr1t"
server_key: "b3a493a49962ffd3a12a201ad66ebebe"
server_key_iv: "b317dca1ff3a371fd5ed589d3ae5b5aa"
server_key_version: "encryptionkey@0"
I20260806 22:41:43.431322 443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "cd37baa871ee4ebbb82552241b0069b2"
format_stamp: "Formatted at 2026-08-06 22:41:43 on dist-test-slave-zr1t"
server_key: "b3a493a49962ffd3a12a201ad66ebebe"
server_key_iv: "b317dca1ff3a371fd5ed589d3ae5b5aa"
server_key_version: "encryptionkey@0"
I20260806 22:41:43.436461 443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:41:43.439973 473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:43.441255 443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 22:41:43.441471 443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "cd37baa871ee4ebbb82552241b0069b2"
format_stamp: "Formatted at 2026-08-06 22:41:43 on dist-test-slave-zr1t"
server_key: "b3a493a49962ffd3a12a201ad66ebebe"
server_key_iv: "b317dca1ff3a371fd5ed589d3ae5b5aa"
server_key_version: "encryptionkey@0"
I20260806 22:41:43.441758 443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:43.460574 443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:43.465054 443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:43.465356 443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:43.466423 443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:43.467800 443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:43.467906 443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:43.468056 443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:43.468129 443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:43.502974 443 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:34075
I20260806 22:41:43.503024 652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:34075 every 8 connection(s)
I20260806 22:41:43.504343 443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:41:43.513151 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 443
I20260806 22:41:43.513401 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:41:43.514088 23954 external_mini_cluster.cc:1428] Setting key 998eb98eb348d5f98b000a30fc449494
Aug 06 22:41:43 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056103, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:43.527866 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:43.507114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:49729 (local address 127.23.100.190:45503)
0806 22:41:43.507570 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:43.507575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:43.508511 (+ 936us) server_negotiation.cc:407] Connection header received
0806 22:41:43.510442 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:43.510447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:43.510541 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:43.510666 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:43.512155 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:43.513094 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:43.515076 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:43.515805 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:43.515976 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:43.520448 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:43.520458 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:43.520465 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:43.520512 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:43.524391 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:43.525216 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:43.525224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:43.525230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:43.525390 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:43.525876 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:43.525883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:43.525889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:43.526274 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:43.526433 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:43.527127 (+ 694us) server_negotiation.cc:299] Negotiation successful
0806 22:41:43.527391 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":348,"thread_start_us":247,"threads_started":1}
I20260806 22:41:43.528901 658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:43.507608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.129:49729)
0806 22:41:43.508330 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:43.508393 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:43.510166 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:43.510902 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:43.510910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:43.511432 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:43.511983 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:43.512000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:43.513325 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:43.513331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:43.514934 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:43.514949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:43.516304 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:43.516312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:43.516603 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:43.517346 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:43.517390 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:43.520172 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:43.524561 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:43.524568 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:43.524582 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:43.525064 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:43.525502 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:43.525505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:43.525507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:43.525748 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:43.526400 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:43.526406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:43.526745 (+ 339us) client_negotiation.cc:769] Sending connection context
0806 22:41:43.526984 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 22:41:43.527386 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":279,"threads_started":1}
I20260806 22:41:43.530637 653 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45503
I20260806 22:41:43.530968 653 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:43.531747 653 heartbeater.cc:511] Master 127.23.100.190:45503 requested a full tablet report, sending...
I20260806 22:41:43.533980 359 ts_manager.cc:194] Registered new tserver with Master: cd37baa871ee4ebbb82552241b0069b2 (127.23.100.129:34075)
I20260806 22:41:43.535468 359 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:49729
WARNING: no policy specified for oryx/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab.
Entry for principal oryx/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab.
I20260806 22:41:43.574549 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45503
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab
--principal=oryx/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:43.911149 661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:43.911494 661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:43.911671 661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:43.919232 661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:43.919399 661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:43.919582 661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:41:43.930109 661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab
--principal=oryx/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:43.931578 661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:43.933040 661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:43.946805 668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:43.946772 669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:43.946830 672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:43.948052 661 server_base.cc:1034] running on GCE node
I20260806 22:41:43.948756 661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:43.949780 661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:43.951023 661 hybrid_clock.cc:648] HybridClock initialized: now 1786056103950967 us; error 66 us; skew 500 ppm
I20260806 22:41:43.951356 661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:43 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056103, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:43.954774 661 init.cc:377] Logged in from keytab as oryx/127.23.100.130@KRBTEST.COM (short username oryx)
I20260806 22:41:43.956367 661 webserver.cc:460] Webserver started at http://127.23.100.130:42979/ using document root <none> and password file <none>
I20260806 22:41:43.957039 661 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:43.957149 661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:43.957406 661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:43.960438 661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "0a62863097244864bed64a9809dfdcde"
format_stamp: "Formatted at 2026-08-06 22:41:43 on dist-test-slave-zr1t"
server_key: "486382038629eae31ee863011d6cf5df"
server_key_iv: "15fe9b747f9b05f48851ce7e33961770"
server_key_version: "encryptionkey@0"
I20260806 22:41:43.961202 661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "0a62863097244864bed64a9809dfdcde"
format_stamp: "Formatted at 2026-08-06 22:41:43 on dist-test-slave-zr1t"
server_key: "486382038629eae31ee863011d6cf5df"
server_key_iv: "15fe9b747f9b05f48851ce7e33961770"
server_key_version: "encryptionkey@0"
I20260806 22:41:43.966674 661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 22:41:43.970268 686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:43.971612 661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:43.971796 661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "0a62863097244864bed64a9809dfdcde"
format_stamp: "Formatted at 2026-08-06 22:41:43 on dist-test-slave-zr1t"
server_key: "486382038629eae31ee863011d6cf5df"
server_key_iv: "15fe9b747f9b05f48851ce7e33961770"
server_key_version: "encryptionkey@0"
I20260806 22:41:43.972010 661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:43.997501 661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:44.001612 661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:44.001909 661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:44.002900 661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:44.004240 661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:44.004314 661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:44.004496 661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:44.004554 661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:44.039696 661 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:46003
I20260806 22:41:44.039747 864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:46003 every 8 connection(s)
I20260806 22:41:44.041109 661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:41:44.050774 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 661
I20260806 22:41:44.050974 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:41:44.051594 23954 external_mini_cluster.cc:1428] Setting key 6249a829ac03c0c934c2492b3746dff5
Aug 06 22:41:44 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056103, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:44.062752 867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:44.044024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:51459 (local address 127.23.100.190:45503)
0806 22:41:44.044522 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:44.044529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:44.045488 (+ 959us) server_negotiation.cc:407] Connection header received
0806 22:41:44.047464 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:44.047468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:44.047537 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:44.047637 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:44.049074 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:44.049899 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:44.052267 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:44.052878 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:44.053012 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:44.056578 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:44.056583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:44.056587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:44.056622 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:44.059422 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:44.060221 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:44.060225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:44.060228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:44.060373 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:44.060854 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:44.060860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:44.060863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:44.061140 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:44.061304 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:44.062089 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 22:41:44.062366 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":238,"threads_started":1}
I20260806 22:41:44.064020 868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:44.044512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.130:51459)
0806 22:41:44.045306 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:44.045350 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:44.047215 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:44.047829 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:44.047836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:44.048350 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:44.048911 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:44.048927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:44.050053 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:44.050058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:44.052062 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:44.052085 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:44.053092 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:44.053097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:44.053299 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:44.054073 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:44.054097 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:44.056336 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:44.059590 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:44.059600 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:44.059616 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:44.060073 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:44.060505 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:44.060510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:44.060514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:44.060714 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:44.061279 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:44.061287 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:44.061712 (+ 425us) client_negotiation.cc:769] Sending connection context
0806 22:41:44.062005 (+ 293us) client_negotiation.cc:240] Negotiation successful
0806 22:41:44.062591 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":282,"threads_started":1}
I20260806 22:41:44.065621 865 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45503
I20260806 22:41:44.065975 865 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:44.066856 865 heartbeater.cc:511] Master 127.23.100.190:45503 requested a full tablet report, sending...
I20260806 22:41:44.068578 359 ts_manager.cc:194] Registered new tserver with Master: 0a62863097244864bed64a9809dfdcde (127.23.100.130:46003)
I20260806 22:41:44.069907 359 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:51459
WARNING: no policy specified for oryx/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab.
Entry for principal oryx/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab.
I20260806 22:41:44.113653 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45503
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab
--principal=oryx/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:44.464238 875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:44.464548 875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:44.464645 875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:44.472179 875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:44.472302 875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:44.472425 875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:41:44.482652 875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab
--principal=oryx/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:45503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:44.483911 875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:44.485178 875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:44.498819 881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:44.498775 882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:44.498720 884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:44.499665 875 server_base.cc:1034] running on GCE node
I20260806 22:41:44.500267 875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:44.501199 875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:44.502512 875 hybrid_clock.cc:648] HybridClock initialized: now 1786056104502455 us; error 56 us; skew 500 ppm
I20260806 22:41:44.502911 875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:44 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056104, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:44.506999 875 init.cc:377] Logged in from keytab as oryx/127.23.100.131@KRBTEST.COM (short username oryx)
I20260806 22:41:44.509337 875 webserver.cc:460] Webserver started at http://127.23.100.131:44849/ using document root <none> and password file <none>
I20260806 22:41:44.510070 875 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:44.510243 875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:44.510552 875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:44.513378 875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "9517100acaf74cc3a6b7caf88c305b03"
format_stamp: "Formatted at 2026-08-06 22:41:44 on dist-test-slave-zr1t"
server_key: "dd18004417caaf023d1260f54a79405f"
server_key_iv: "efc353fd800452c1622853c6f08ff94e"
server_key_version: "encryptionkey@0"
I20260806 22:41:44.514266 875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "9517100acaf74cc3a6b7caf88c305b03"
format_stamp: "Formatted at 2026-08-06 22:41:44 on dist-test-slave-zr1t"
server_key: "dd18004417caaf023d1260f54a79405f"
server_key_iv: "efc353fd800452c1622853c6f08ff94e"
server_key_version: "encryptionkey@0"
I20260806 22:41:44.519615 875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:41:44.523558 895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:44.525017 875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 22:41:44.525204 875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "9517100acaf74cc3a6b7caf88c305b03"
format_stamp: "Formatted at 2026-08-06 22:41:44 on dist-test-slave-zr1t"
server_key: "dd18004417caaf023d1260f54a79405f"
server_key_iv: "efc353fd800452c1622853c6f08ff94e"
server_key_version: "encryptionkey@0"
I20260806 22:41:44.525415 875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:44.538687 653 heartbeater.cc:503] Master 127.23.100.190:45503 was elected leader, sending a full tablet report...
I20260806 22:41:44.539788 875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:44.544521 875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:44.544772 875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:44.545949 875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:44.547326 875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:44.547405 875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:44.547586 875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:44.547662 875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:44.580955 875 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:34259
I20260806 22:41:44.580998 1075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:34259 every 8 connection(s)
I20260806 22:41:44.582428 875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:41:44.592082 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 875
I20260806 22:41:44.592283 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:41:44.592773 23954 external_mini_cluster.cc:1428] Setting key f7322a6e3de0852817384adf60536a75
Aug 06 22:41:44 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056104, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:44.603286 1079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:44.585084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:49427 (local address 127.23.100.190:45503)
0806 22:41:44.585625 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:44.585630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:44.586580 (+ 950us) server_negotiation.cc:407] Connection header received
0806 22:41:44.588463 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:44.588469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:44.588562 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:44.588686 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:44.590291 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:44.591087 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:44.592765 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:44.593236 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:44.593402 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:44.597347 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:44.597353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:44.597356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:44.597393 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:44.600109 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:44.600874 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:44.600879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:44.600881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:44.601006 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:44.601466 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:44.601471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:44.601473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:44.601799 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:44.601964 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:44.602665 (+ 701us) server_negotiation.cc:299] Negotiation successful
0806 22:41:44.602887 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":279,"threads_started":1}
I20260806 22:41:44.604295 1080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:44.585539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.131:49427)
0806 22:41:44.586301 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:44.586361 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:44.588172 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:44.588940 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:44.588949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:44.589470 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:44.590113 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:44.590130 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:44.591268 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:44.591272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:44.592611 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:44.592624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:44.593498 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:44.593504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:44.593870 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:44.594700 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:44.594736 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:44.597137 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:44.600271 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:44.600280 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:44.600296 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:44.600732 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:44.601124 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:44.601129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:44.601133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:44.601332 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:44.601939 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:44.601947 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:44.602301 (+ 354us) client_negotiation.cc:769] Sending connection context
0806 22:41:44.602574 (+ 273us) client_negotiation.cc:240] Negotiation successful
0806 22:41:44.603118 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":293,"threads_started":1}
I20260806 22:41:44.605746 1076 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45503
I20260806 22:41:44.606025 1076 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:44.606756 1076 heartbeater.cc:511] Master 127.23.100.190:45503 requested a full tablet report, sending...
I20260806 22:41:44.608271 357 ts_manager.cc:194] Registered new tserver with Master: 9517100acaf74cc3a6b7caf88c305b03 (127.23.100.131:34259)
I20260806 22:41:44.609112 357 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:49427
I20260806 22:41:44.620370 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.100.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.189.keytab.
Entry for principal oryx/127.23.100.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.189.keytab.
I20260806 22:41:44.678105 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.189:37055
--webserver_interface=127.23.100.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.189.keytab
--principal=oryx/127.23.100.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:45503,127.23.100.189:37055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:45.028406 1086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:45.028720 1086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:45.028816 1086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:45.036602 1086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:45.036715 1086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:45.036764 1086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:45.036823 1086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:45.036866 1086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:45.050307 1086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:45503,127.23.100.189:37055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.189.keytab
--principal=oryx/127.23.100.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.189:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.100.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
W20260806 22:41:45.050765 1086 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.100.190:45503,127.23.100.189:37055'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 22:41:45.052248 1086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:45.053966 1086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:45.065923 1094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:45.066006 1097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:45.065974 1095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:45.066835 1086 server_base.cc:1034] running on GCE node
I20260806 22:41:45.067698 1086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:45.069213 1086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:45.070513 1086 hybrid_clock.cc:648] HybridClock initialized: now 1786056105070411 us; error 61 us; skew 500 ppm
I20260806 22:41:45.070892 1086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:41:45.073602 865 heartbeater.cc:503] Master 127.23.100.190:45503 was elected leader, sending a full tablet report...
Aug 06 22:41:45 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056105, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:45.074595 1086 init.cc:377] Logged in from keytab as oryx/127.23.100.189@KRBTEST.COM (short username oryx)
I20260806 22:41:45.076467 1086 webserver.cc:460] Webserver started at http://127.23.100.189:38315/ using document root <none> and password file <none>
I20260806 22:41:45.077241 1086 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:45.077383 1086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:45.077775 1086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:45.080736 1086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data/instance:
uuid: "e04600f32f8c41ed812bcc5bd9332e28"
format_stamp: "Formatted at 2026-08-06 22:41:45 on dist-test-slave-zr1t"
server_key: "7d60cb6b671f4b5f44feb32b924ad4d0"
server_key_iv: "8bf64f070e246b90f28001ed4eb67cb9"
server_key_version: "encryptionkey@0"
I20260806 22:41:45.081594 1086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/wal/instance:
uuid: "e04600f32f8c41ed812bcc5bd9332e28"
format_stamp: "Formatted at 2026-08-06 22:41:45 on dist-test-slave-zr1t"
server_key: "7d60cb6b671f4b5f44feb32b924ad4d0"
server_key_iv: "8bf64f070e246b90f28001ed4eb67cb9"
server_key_version: "encryptionkey@0"
I20260806 22:41:45.086874 1086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:41:45.090379 1107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:45.091759 1086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:45.091954 1086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/wal
uuid: "e04600f32f8c41ed812bcc5bd9332e28"
format_stamp: "Formatted at 2026-08-06 22:41:45 on dist-test-slave-zr1t"
server_key: "7d60cb6b671f4b5f44feb32b924ad4d0"
server_key_iv: "8bf64f070e246b90f28001ed4eb67cb9"
server_key_version: "encryptionkey@0"
I20260806 22:41:45.092186 1086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:45.106345 1086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:45.110580 1086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:45.110860 1086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:45.130194 1086 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.189:37055
I20260806 22:41:45.130214 1191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.189:37055 every 8 connection(s)
I20260806 22:41:45.131773 1086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
I20260806 22:41:45.134696 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 1086
I20260806 22:41:45.136130 1192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:45.147171 1192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:41:45 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.189@KRBTEST.COM
Aug 06 22:41:45 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056105, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.189@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:45.160521 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.136784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.23.100.189:37055)
0806 22:41:45.137643 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.137660 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.137719 (+ 59us) server_negotiation.cc:407] Connection header received
0806 22:41:45.139967 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.140000 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.140576 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.141078 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.142662 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.144993 (+ 2331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.146559 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.147762 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.147967 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.153043 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:45.153054 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:45.153064 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:45.153101 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:45.156233 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.156989 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.156997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.157004 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.157170 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.157647 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.157659 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.157662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.158224 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:45.158592 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.159041 (+ 449us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.159509 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":345,"threads_started":1}
I20260806 22:41:45.165892 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.150467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.23.100.190:45503)
0806 22:41:45.151070 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.151077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.151650 (+ 573us) server_negotiation.cc:407] Connection header received
0806 22:41:45.151854 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.151858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.151923 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.152049 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.153240 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.154262 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.155907 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.156317 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.156470 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.159940 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:45.159945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:45.159954 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:45.159994 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:45.162829 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.163703 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.163709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.163713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.163849 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.164242 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.164247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.164250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.164610 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:45.164773 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.165239 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.165478 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":309,"threads_started":1}
I20260806 22:41:45.165939 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.150450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:52704)
0806 22:41:45.151471 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.151524 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:45.151718 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:45.152115 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:45.152123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:45.152764 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:45.153074 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.153084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.154415 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.154421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.155723 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.155747 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.156505 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.156512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.156704 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.157318 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:45.157335 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:45.159742 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:45.162973 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.162988 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.162995 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.163578 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.163964 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.163971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.163975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.164132 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.164745 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:45.164753 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:45.164939 (+ 186us) client_negotiation.cc:769] Sending connection context
0806 22:41:45.165134 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 22:41:45.165479 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":551,"threads_started":1}
I20260806 22:41:45.167932 1192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:41:45 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056105, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.189@KRBTEST.COM for oryx/127.23.100.189@KRBTEST.COM
I20260806 22:41:45.180218 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.168554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:37055 (local address 127.0.0.1:37184)
0806 22:41:45.168728 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.168744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:45.168871 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:45.169154 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:45.169158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:45.169631 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:45.169776 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.169785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.170752 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.170757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.171619 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.171630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.172341 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.172346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.172525 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.173035 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:45.173052 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:45.175183 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:45.178070 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.178074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.178076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.178437 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.178848 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.178852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.178855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.178991 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.179488 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:45.179492 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:45.179625 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 22:41:45.179740 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 22:41:45.179973 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 22:41:45.180595 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.168748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.23.100.189:37055)
0806 22:41:45.168908 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.168916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.168935 (+ 19us) server_negotiation.cc:407] Connection header received
0806 22:41:45.168977 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.168982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.169038 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.169139 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.169908 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.170610 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.171760 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.172183 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.172308 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.175340 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:45.175345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:45.175348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:45.175376 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:45.177946 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.178555 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.178559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.178561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.178708 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.179101 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.179104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.179106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.179375 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:45.179483 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.179933 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.180127 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 22:41:45.185328 1192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: Bootstrap starting.
I20260806 22:41:45.188656 1192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:45.189934 1192 log.cc:824] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:45.192952 1192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: No bootstrap required, opened a new log
I20260806 22:41:45.200500 1192 raft_consensus.cc:348] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } }
I20260806 22:41:45.200852 1192 raft_consensus.cc:374] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:45.200960 1192 raft_consensus.cc:729] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04600f32f8c41ed812bcc5bd9332e28, State: Initialized, Role: FOLLOWER
I20260806 22:41:45.201706 1192 consensus_queue.cc:260] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } }
I20260806 22:41:45.202636 1203 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } } }
I20260806 22:41:45.203223 1203 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:41:45.203745 1192 sys_catalog.cc:564] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 22:41:45.213330 1221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 22:41:45.213594 1221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 22:41:45.216408 1086 master_runner.cc:428] Detected that this master e04600f32f8c41ed812bcc5bd9332e28 is joining an existing cluster
I20260806 22:41:45.216524 1086 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.100.189:37055
I20260806 22:41:45.243551 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.222522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.23.100.189:37055)
0806 22:41:45.222714 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.222718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.226858 (+ 4140us) server_negotiation.cc:407] Connection header received
0806 22:41:45.227108 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.227120 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.227205 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.227309 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.229130 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.230135 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.232303 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.233003 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.233096 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.236072 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:45.236079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:45.236085 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:45.236122 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:45.240187 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.240968 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.240975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.240978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.241105 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.241645 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.241650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.241653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.242003 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:45.242124 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.242761 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.243155 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":183,"server-negotiator.queue_time_us":50}
I20260806 22:41:45.243575 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.222215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:52708)
0806 22:41:45.226857 (+ 4642us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.226878 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:45.227044 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:45.227629 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:45.227635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:45.228571 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:45.229084 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.229117 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.230815 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.230820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.232562 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.232575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.233895 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.233904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.234069 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.234698 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:45.234719 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:45.235863 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:45.240187 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.240193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.240197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.240747 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.241239 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.241243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.241247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.241430 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.242205 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:45.242209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:45.242472 (+ 263us) client_negotiation.cc:769] Sending connection context
0806 22:41:45.242637 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 22:41:45.243021 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4496,"mutex_wait_us":5,"spinlock_wait_cycles":1408,"thread_start_us":6397,"threads_started":1}
I20260806 22:41:45.244376 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.221637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.23.100.190:45503)
0806 22:41:45.221891 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.221900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.226988 (+ 5088us) server_negotiation.cc:407] Connection header received
0806 22:41:45.227199 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.227206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.227312 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.227432 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.229354 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.230639 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.232748 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.233442 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.233613 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.236103 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:45.236109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:45.236114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:45.236155 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:45.240000 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.240908 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.240915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.240919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.241107 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.241588 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.241595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.241600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.241991 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:45.242125 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.243752 (+ 1627us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.244020 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 22:41:45.245483 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.222105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:37055 (local address 127.0.0.1:37188)
0806 22:41:45.226705 (+ 4600us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.226738 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:45.226926 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:45.227509 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:45.227515 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:45.228380 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:45.228961 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.228979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.230296 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.230302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.232155 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.232171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.233429 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.233436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.233646 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.234400 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:45.234417 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:45.235863 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:45.240329 (+ 4466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.240337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.240341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.240862 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.241240 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.241243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.241247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.241535 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.242205 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:45.242209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:45.242468 (+ 259us) client_negotiation.cc:769] Sending connection context
0806 22:41:45.242846 (+ 378us) client_negotiation.cc:240] Negotiation successful
0806 22:41:45.244979 (+ 2133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4405,"mutex_wait_us":1822,"spinlock_wait_cycles":78208,"thread_start_us":3774,"threads_started":1}
WARNING: no policy specified for oryx/127.23.100.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.188@KRBTEST.COM" created.
Aug 06 22:41:45 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056102, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.190@KRBTEST.COM for oryx/127.23.100.189@KRBTEST.COM
I20260806 22:41:45.267938 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.251209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.23.100.189:37055)
0806 22:41:45.251407 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.251416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.252079 (+ 663us) server_negotiation.cc:407] Connection header received
0806 22:41:45.252463 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.252469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.252548 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.252665 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.254037 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.254856 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.256511 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.257010 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.257158 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.261154 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:45.261165 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:45.261168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:45.261202 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:45.264362 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.265268 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.265276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.265280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.265449 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.265921 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.265927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.265931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.266323 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:45.266483 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.267185 (+ 702us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.267474 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 22:41:45.269114 1235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.251203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:37055 (local address 127.0.0.1:37192)
0806 22:41:45.251940 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.251977 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:45.252213 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:45.252726 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:45.252734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:45.253474 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:45.253856 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.253874 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.255001 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.255008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.256351 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.256366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.257188 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.257194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.257398 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.258163 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:45.258183 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:45.260929 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:45.264560 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.264580 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.264589 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.265117 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.265617 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.265622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.265626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.265792 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.266572 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:45.266584 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:45.266834 (+ 250us) client_negotiation.cc:769] Sending connection context
0806 22:41:45.268477 (+ 1643us) client_negotiation.cc:240] Negotiation successful
0806 22:41:45.268790 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":370,"threads_started":1}
I20260806 22:41:45.271239 357 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.100.189:37055: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28"
member_type: NON_VOTER
last_known_addr {
host: "127.23.100.189"
port: 37055
}
attrs {
promote: true
}
}
dest_uuid: "9a9e131374e743a4803b662ca72cf6dd"
cas_config_opid_index: -1
I20260806 22:41:45.272246 357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } }
I20260806 22:41:45.288887 1155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:45.293080 1242 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e04600f32f8c41ed812bcc5bd9332e28 (127.23.100.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } } }
I20260806 22:41:45.295120 1155 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 FOLLOWER]: Refusing update from remote peer 9a9e131374e743a4803b662ca72cf6dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.188.keytab.
Entry for principal oryx/127.23.100.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.188.keytab.
I20260806 22:41:45.297627 1242 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.100.189:37055
I20260806 22:41:45.298687 1086 master.cc:537] Master@127.23.100.189:37055 shutting down...
I20260806 22:41:45.298729 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.188:46107
--webserver_interface=127.23.100.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.188.keytab
--principal=oryx/127.23.100.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:45503,127.23.100.189:37055,127.23.100.188:46107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:41:45.296873 1239 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.013s
I20260806 22:41:45.298440 1244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } } }
I20260806 22:41:45.310400 1244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:45.312848 1086 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:45.313588 1086 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 22:41:45.313781 333 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260806 22:41:45.313949 1086 tablet_replica.cc:330] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: stopping tablet replica
I20260806 22:41:45.321336 1086 master.cc:552] Master@127.23.100.189:37055 shutdown complete.
I20260806 22:41:45.321547 1086 master_runner.cc:305] Clearing existing system tablet
I20260806 22:41:45.322099 1086 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:45.322256 1086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:45.325824 1086 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:41:45.327049 1235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.326512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:37055 (local address 127.0.0.1:37204)
0806 22:41:45.326707 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.326844 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:37055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 22:41:45.327620 333 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd -> Peer e04600f32f8c41ed812bcc5bd9332e28 (127.23.100.189:37055): Couldn't send request to peer e04600f32f8c41ed812bcc5bd9332e28. Status: Network error: Client connection negotiation failed: client connection to 127.23.100.189:37055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 22:41:45.334002 1250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:45.334458 1086 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:45.334795 1086 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.007s
I20260806 22:41:45.334950 1086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/wal
uuid: "e04600f32f8c41ed812bcc5bd9332e28"
format_stamp: "Formatted at 2026-08-06 22:41:45 on dist-test-slave-zr1t"
server_key: "7d60cb6b671f4b5f44feb32b924ad4d0"
server_key_iv: "8bf64f070e246b90f28001ed4eb67cb9"
server_key_version: "encryptionkey@0"
I20260806 22:41:45.340829 1086 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:45.349088 1086 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 22:41:45.349351 1086 log.cc:1196] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 22:41:45.349901 1086 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: Deleting consensus metadata
I20260806 22:41:45.350234 1086 master_runner.cc:315] Copying system tablet from 127.23.100.190:45503
I20260806 22:41:45.353188 1086 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: tablet copy: Beginning tablet copy session from remote peer at address 127.23.100.190:45503
I20260806 22:41:45.373291 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.354391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.23.100.190:45503)
0806 22:41:45.354656 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.354667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.357307 (+ 2640us) server_negotiation.cc:407] Connection header received
0806 22:41:45.357578 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.357584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.357679 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.357795 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.359181 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.360282 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.362159 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.362980 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.363147 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.365629 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:45.365639 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:45.365644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:45.365696 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:45.369796 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.370690 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.370697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.370701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.370874 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.371343 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.371349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.371353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.371822 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:45.372008 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.372570 (+ 562us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.372846 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 22:41:45.373335 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.354178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:52720)
0806 22:41:45.357152 (+ 2974us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.357176 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:45.357382 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:45.358018 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:45.358023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:45.358619 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:45.358943 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.358959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.360491 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.360496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.361986 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.362002 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.363184 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.363190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.363373 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.364195 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:45.364219 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:45.365340 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:45.369964 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.369970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.369974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.370538 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.371016 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.371025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.371029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.371201 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.371984 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:45.371989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:45.372214 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 22:41:45.372439 (+ 225us) client_negotiation.cc:240] Negotiation successful
0806 22:41:45.372850 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2806,"thread_start_us":2680,"threads_started":1}
I20260806 22:41:45.374982 383 tablet_copy_service.cc:140] P 9a9e131374e743a4803b662ca72cf6dd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e04600f32f8c41ed812bcc5bd9332e28 ({username='oryx', principal='oryx/127.23.100.189@KRBTEST.COM'} at 127.0.0.1:52720)
I20260806 22:41:45.375348 383 tablet_copy_service.cc:161] P 9a9e131374e743a4803b662ca72cf6dd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e04600f32f8c41ed812bcc5bd9332e28 at {username='oryx', principal='oryx/127.23.100.189@KRBTEST.COM'} at 127.0.0.1:52720: session id = e04600f32f8c41ed812bcc5bd9332e28-00000000000000000000000000000000
I20260806 22:41:45.380627 383 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 22:41:45.383531 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 22:41:45.391346 1086 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: tablet copy: Starting download of 0 data blocks...
I20260806 22:41:45.391721 1086 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: tablet copy: Starting download of 1 WAL segments...
I20260806 22:41:45.395088 1086 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 22:41:45.400040 383 tablet_copy_service.cc:342] P 9a9e131374e743a4803b662ca72cf6dd: Request end of tablet copy session e04600f32f8c41ed812bcc5bd9332e28-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.100.189@KRBTEST.COM'} at 127.0.0.1:52720
I20260806 22:41:45.400306 383 tablet_copy_service.cc:433] P 9a9e131374e743a4803b662ca72cf6dd: ending tablet copy session e04600f32f8c41ed812bcc5bd9332e28-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e04600f32f8c41ed812bcc5bd9332e28
I20260806 22:41:45.413796 1086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:45.416826 1259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:45.418591 1260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:45.431617 1264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:45.433990 1086 server_base.cc:1034] running on GCE node
I20260806 22:41:45.434456 1086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:45.435199 1086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:45.436589 1086 hybrid_clock.cc:648] HybridClock initialized: now 1786056105436505 us; error 43 us; skew 500 ppm
I20260806 22:41:45.436848 1086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:45 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056105, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:45.438933 1086 init.cc:377] Logged in from keytab as oryx/127.23.100.189@KRBTEST.COM (short username oryx)
I20260806 22:41:45.440348 1086 webserver.cc:460] Webserver started at http://127.23.100.189:36813/ using document root <none> and password file <none>
I20260806 22:41:45.440752 1086 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:45.440860 1086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:45.443449 1086 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260806 22:41:45.445220 1274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:45.445744 1086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260806 22:41:45.445967 1086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/wal
uuid: "e04600f32f8c41ed812bcc5bd9332e28"
format_stamp: "Formatted at 2026-08-06 22:41:45 on dist-test-slave-zr1t"
server_key: "7d60cb6b671f4b5f44feb32b924ad4d0"
server_key_iv: "8bf64f070e246b90f28001ed4eb67cb9"
server_key_version: "encryptionkey@0"
I20260806 22:41:45.446123 1086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:45.462733 1086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:45.466311 1086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:45.498358 1086 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.189:37055
I20260806 22:41:45.498380 1347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.189:37055 every 8 connection(s)
I20260806 22:41:45.499413 1086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
I20260806 22:41:45.512370 1349 sys_catalog.cc:263] Verifying existing consensus state
I20260806 22:41:45.516412 1349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: Bootstrap starting.
I20260806 22:41:45.535935 1349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 22:41:45.536645 1349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: Bootstrap complete.
I20260806 22:41:45.537892 1349 raft_consensus.cc:348] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } }
I20260806 22:41:45.538260 1349 raft_consensus.cc:729] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e04600f32f8c41ed812bcc5bd9332e28, State: Initialized, Role: LEARNER
I20260806 22:41:45.538568 1349 consensus_queue.cc:260] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } }
I20260806 22:41:45.539913 1349 sys_catalog.cc:564] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:45.539688 1352 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } } }
I20260806 22:41:45.540200 1352 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: This master's current role is: LEARNER
I20260806 22:41:45.550647 1373 catalog_manager.cc:1259] Loaded cluster ID: 9743b2c273644d7899b4f95bc4699be9
I20260806 22:41:45.551143 1373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: loading cluster ID for follower catalog manager: success
I20260806 22:41:45.554699 1373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: acquiring CA information for follower catalog manager: success
I20260806 22:41:45.557300 1373 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 22:41:45.611601 1076 heartbeater.cc:503] Master 127.23.100.190:45503 was elected leader, sending a full tablet report...
W20260806 22:41:45.733425 1248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:45.733803 1248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:45.733901 1248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:45.741637 1248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:45.741748 1248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:45.741796 1248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:45.741855 1248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:45.741907 1248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:45.752631 1248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:45503,127.23.100.189:37055,127.23.100.188:46107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.188.keytab
--principal=oryx/127.23.100.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.188:46107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.100.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:45.753974 1248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:45.755223 1248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:45.767434 1380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:45.767408 1383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:45.767486 1379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:45.768548 1248 server_base.cc:1034] running on GCE node
I20260806 22:41:45.769372 1248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:45.770895 1248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:45.772100 1248 hybrid_clock.cc:648] HybridClock initialized: now 1786056105772063 us; error 35 us; skew 500 ppm
I20260806 22:41:45.772480 1248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:45 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056105, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:45.780635 1248 init.cc:377] Logged in from keytab as oryx/127.23.100.188@KRBTEST.COM (short username oryx)
I20260806 22:41:45.782344 1248 webserver.cc:460] Webserver started at http://127.23.100.188:39679/ using document root <none> and password file <none>
I20260806 22:41:45.783069 1248 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:45.783223 1248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:45.783547 1248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:45.786651 1248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data/instance:
uuid: "a2153cc3e61143849fa047c85042a332"
format_stamp: "Formatted at 2026-08-06 22:41:45 on dist-test-slave-zr1t"
server_key: "8b8f71fb3b038e3f5efe8af0f574aa34"
server_key_iv: "5014bd0bee8b1aa8fdd0253d7724f13c"
server_key_version: "encryptionkey@0"
I20260806 22:41:45.787451 1248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/wal/instance:
uuid: "a2153cc3e61143849fa047c85042a332"
format_stamp: "Formatted at 2026-08-06 22:41:45 on dist-test-slave-zr1t"
server_key: "8b8f71fb3b038e3f5efe8af0f574aa34"
server_key_iv: "5014bd0bee8b1aa8fdd0253d7724f13c"
server_key_version: "encryptionkey@0"
I20260806 22:41:45.792891 1248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 22:41:45.796428 1397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:45.797891 1248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:41:45.798076 1248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/wal
uuid: "a2153cc3e61143849fa047c85042a332"
format_stamp: "Formatted at 2026-08-06 22:41:45 on dist-test-slave-zr1t"
server_key: "8b8f71fb3b038e3f5efe8af0f574aa34"
server_key_iv: "5014bd0bee8b1aa8fdd0253d7724f13c"
server_key_version: "encryptionkey@0"
I20260806 22:41:45.798302 1248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:45.837402 1248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:45.844143 1248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:45.844491 1248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:45.884019 1248 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.188:46107
I20260806 22:41:45.884038 1470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.188:46107 every 8 connection(s)
I20260806 22:41:45.886132 1248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
I20260806 22:41:45.890988 1473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:45.891119 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 1248
I20260806 22:41:45.902277 1473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:41:45 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.188@KRBTEST.COM
Aug 06 22:41:45 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056105, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.188@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:45.919028 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.893568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.23.100.188:46107)
0806 22:41:45.894561 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.894580 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.894643 (+ 63us) server_negotiation.cc:407] Connection header received
0806 22:41:45.897584 (+ 2941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.897619 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.898248 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.900856 (+ 2608us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.901307 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.902699 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.904173 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.904840 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.905063 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.910496 (+ 5433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:45.910504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:45.910518 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:45.910566 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:45.914063 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.914851 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.914862 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.914879 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.915106 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.915496 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.915502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.915506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.916274 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:45.916593 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.917076 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.917570 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":392,"threads_started":1}
I20260806 22:41:45.922086 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.905078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.23.100.190:45503)
0806 22:41:45.905666 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.905673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.906241 (+ 568us) server_negotiation.cc:407] Connection header received
0806 22:41:45.906420 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.906426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.906515 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.906644 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.907676 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.908763 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.910824 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.911463 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.911598 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.915028 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:45.915034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:45.915038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:45.915079 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:45.918595 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.919639 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.919644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.919646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.919777 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.920242 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.920246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.920248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.920583 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:45.920833 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.921433 (+ 600us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.921797 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":294,"threads_started":1}
I20260806 22:41:45.922145 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.905098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:52732)
0806 22:41:45.906079 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.906119 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:45.906286 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:45.906706 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:45.906711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:45.907258 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:45.907516 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.907524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.908965 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.908974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.910676 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.910689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.911658 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.911663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.911802 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.912472 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:45.912501 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:45.914841 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:45.918758 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.918769 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.918774 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.919494 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.919911 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.919916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.919919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.920091 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.920820 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:45.920829 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:45.921044 (+ 215us) client_negotiation.cc:769] Sending connection context
0806 22:41:45.921288 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 22:41:45.921670 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":616,"threads_started":1}
I20260806 22:41:45.923482 1473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:41:45 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056105, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.188@KRBTEST.COM for oryx/127.23.100.189@KRBTEST.COM
I20260806 22:41:45.934916 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.928002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:37055 (local address 127.0.0.1:37220)
0806 22:41:45.928529 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.928549 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:45.928672 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:45.929171 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:45.929177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:45.929218 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:45.929595 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.929607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.931035 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.931040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.932433 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.932447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.934036 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.934044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.934216 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.934278 (+ 62us) client_negotiation.cc:769] Sending connection context
0806 22:41:45.934392 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 22:41:45.934535 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":280,"threads_started":1}
I20260806 22:41:45.934947 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.928080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37220 (local address 127.23.100.189:37055)
0806 22:41:45.928857 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.928861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.928881 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:41:45.928932 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.928935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.929000 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.929131 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:45.929791 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.930886 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.932585 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.933818 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.934085 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.934311 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.934402 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.934535 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":638,"thread_start_us":495,"threads_started":1}
I20260806 22:41:45.936278 1311 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260806 22:41:45.936928 1352 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a9e131374e743a4803b662ca72cf6dd. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } } }
I20260806 22:41:45.937217 1352 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: This master's current role is: LEARNER
I20260806 22:41:45.938974 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.924168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:37055 (local address 127.0.0.1:37216)
0806 22:41:45.924364 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.924394 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:45.924505 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:45.925166 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:45.925170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:45.925550 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:45.925709 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.925718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.927040 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.927048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.928052 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.928077 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.929320 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.929326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.929560 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.930211 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:45.930229 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:45.932353 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:45.936463 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.936467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.936470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.937072 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.937607 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.937611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.937614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.937754 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.938244 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:45.938248 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:45.938375 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 22:41:45.938508 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 22:41:45.938739 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 22:41:45.939064 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.924316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.23.100.189:37055)
0806 22:41:45.924866 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.924871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.924887 (+ 16us) server_negotiation.cc:407] Connection header received
0806 22:41:45.924938 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.924941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.924998 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.925089 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.925873 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.926873 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.928274 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.929128 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.929333 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.932562 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:45.932567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:45.932571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:45.932603 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:45.936308 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.937239 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.937246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.937250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.937443 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.937856 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.937859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.937861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.938138 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:45.938262 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.938620 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.938783 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":428,"thread_start_us":313,"threads_started":1}
I20260806 22:41:45.940047 1473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:41:45 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056105, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.188@KRBTEST.COM for oryx/127.23.100.188@KRBTEST.COM
I20260806 22:41:45.953248 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.940680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46107 (local address 127.0.0.1:36328)
0806 22:41:45.940835 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.940851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:45.941008 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:45.941235 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:45.941238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:45.941679 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:45.941824 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.941833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.942784 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.942788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.943628 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.943640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.944279 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.944283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:45.944449 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.945093 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:45.945109 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:45.947781 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:45.950810 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.950815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.950819 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.951177 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.951529 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:45.951535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:45.951539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:45.951690 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:45.952201 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:45.952205 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:45.952376 (+ 171us) client_negotiation.cc:769] Sending connection context
0806 22:41:45.952512 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 22:41:45.952785 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 22:41:45.953269 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.940841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.23.100.188:46107)
0806 22:41:45.941009 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.941013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.941028 (+ 15us) server_negotiation.cc:407] Connection header received
0806 22:41:45.941090 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.941093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.941146 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.941216 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.941968 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.942669 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.943748 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:45.944141 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:45.944234 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:45.948000 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:45.948008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:45.948015 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:45.948048 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:45.950662 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.951291 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.951299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.951302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.951416 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:45.951794 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:45.951801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:45.951804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:45.952089 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:45.952239 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:45.952596 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 22:41:45.952937 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":53}
I20260806 22:41:45.958547 1473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Bootstrap starting.
I20260806 22:41:45.961880 1473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:45.962991 1473 log.cc:824] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:45.965746 1473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: No bootstrap required, opened a new log
I20260806 22:41:45.973445 1473 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } }
I20260806 22:41:45.973906 1473 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:45.974069 1473 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2153cc3e61143849fa047c85042a332, State: Initialized, Role: FOLLOWER
I20260806 22:41:45.974740 1473 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } }
I20260806 22:41:45.975788 1489 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } } }
I20260806 22:41:45.976092 1489 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:41:45.976888 1473 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 22:41:45.986097 1505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 22:41:45.986274 1505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 22:41:45.990844 1248 master_runner.cc:428] Detected that this master a2153cc3e61143849fa047c85042a332 is joining an existing cluster
I20260806 22:41:45.991024 1248 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.100.188:46107
I20260806 22:41:46.016993 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.995834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.23.100.189:37055)
0806 22:41:45.995993 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.995997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:45.996809 (+ 812us) server_negotiation.cc:407] Connection header received
0806 22:41:45.997142 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:45.997150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:45.997254 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:45.997358 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:45.999294 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.000769 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.003800 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.004753 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.004893 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.009723 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:46.009731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:46.009740 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:46.009791 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:46.013641 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.014521 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.014527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.014530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.014703 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.015216 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.015224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.015229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.015622 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:46.015772 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:46.016347 (+ 575us) server_negotiation.cc:299] Negotiation successful
0806 22:41:46.016609 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 22:41:46.018358 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.995859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:37055 (local address 127.0.0.1:37230)
0806 22:41:45.996669 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:45.996694 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:45.996958 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:45.997662 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:45.997670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:45.998595 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:45.999073 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:45.999090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.001038 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.001046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.003578 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.003600 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.005499 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.005511 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.005812 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.007528 (+ 1716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:46.007564 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:46.009441 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:46.013801 (+ 4360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.013806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.013810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.014366 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.014859 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.014864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.014869 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.015071 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.015789 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:46.015795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:46.016025 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 22:41:46.017403 (+ 1378us) client_negotiation.cc:240] Negotiation successful
0806 22:41:46.017805 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":424,"threads_started":1}
I20260806 22:41:46.025189 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.996781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46107 (local address 127.0.0.1:36338)
0806 22:41:46.009219 (+ 12438us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:46.009240 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:46.009579 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:46.010064 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:46.010068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:46.010912 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:46.011249 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.011261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.012791 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.012796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.014578 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.014593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.015887 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.015895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.016193 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.016884 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:46.016905 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:46.018337 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:46.022523 (+ 4186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.022530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.022534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.022911 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.023290 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.023294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.023298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.023441 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.024012 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:46.024017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:46.024194 (+ 177us) client_negotiation.cc:769] Sending connection context
0806 22:41:46.024325 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 22:41:46.024626 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12290,"mutex_wait_us":28,"thread_start_us":12579,"threads_started":1}
I20260806 22:41:46.026021 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.995300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.23.100.190:45503)
0806 22:41:45.995522 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.995527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:46.008604 (+ 13077us) server_negotiation.cc:407] Connection header received
0806 22:41:46.009097 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:46.009104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:46.009212 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:46.009294 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:46.011026 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.012240 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.014522 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.015182 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.015279 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.018707 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:46.018713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:46.018718 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:46.018763 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:46.022595 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.023549 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.023555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.023559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.023716 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.024217 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.024223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.024227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.024579 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:46.024724 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:46.025398 (+ 674us) server_negotiation.cc:299] Negotiation successful
0806 22:41:46.025662 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 22:41:46.026173 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.997079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36338 (local address 127.23.100.188:46107)
0806 22:41:45.997254 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:45.997260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:46.009324 (+ 12064us) server_negotiation.cc:407] Connection header received
0806 22:41:46.009718 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:46.009723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:46.009798 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:46.009865 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:46.011407 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.012546 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.014720 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.015468 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.015572 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.018525 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:46.018531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:46.018535 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:46.018567 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:46.022399 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.023031 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.023036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.023039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.023178 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.023560 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.023565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.023568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.023895 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:46.024077 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:46.024431 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 22:41:46.025851 (+ 1420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1198,"server-negotiator.queue_time_us":49}
I20260806 22:41:46.026261 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 443
I20260806 22:41:46.027097 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:45.996055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:52744)
0806 22:41:46.008474 (+ 12419us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:46.008497 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:46.008900 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:46.009503 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:46.009509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:46.010341 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:46.010813 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.010839 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.012462 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.012467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.014368 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.014377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.015708 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.015722 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.016099 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.017086 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:46.017111 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:46.018460 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:46.022778 (+ 4318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.022783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.022787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.023382 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.023882 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.023887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.023890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.024068 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.024814 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:46.024819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:46.025048 (+ 229us) client_negotiation.cc:769] Sending connection context
0806 22:41:46.026221 (+ 1173us) client_negotiation.cc:240] Negotiation successful
0806 22:41:46.026599 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12173,"mutex_wait_us":80,"thread_start_us":11790,"threads_started":1}
Aug 06 22:41:46 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056102, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.190@KRBTEST.COM for oryx/127.23.100.188@KRBTEST.COM
Aug 06 22:41:46 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.129@KRBTEST.COM
I20260806 22:41:46.049218 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.033955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.23.100.188:46107)
0806 22:41:46.034108 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:46.034115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:46.034151 (+ 36us) server_negotiation.cc:407] Connection header received
0806 22:41:46.034387 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:46.034394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:46.034484 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:46.034598 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:46.035343 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.036044 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.037204 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.037756 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.037893 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.042001 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:46.042010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:46.042017 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:46.042056 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:46.046318 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.047183 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.047189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.047193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.047315 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.047728 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.047733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.047736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.048083 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:46.048233 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:46.048641 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 22:41:46.048872 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 22:41:46.049228 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.033897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46107 (local address 127.0.0.1:36344)
0806 22:41:46.034079 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:46.034098 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:46.034223 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:46.034619 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:46.034626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:46.035027 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:46.035181 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.035191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.036190 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.036196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.037066 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.037077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.037900 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.037906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.038088 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.038719 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:46.038749 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:46.041743 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:46.046460 (+ 4717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.046466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.046470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.047016 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.047437 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.047442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.047446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.047611 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.048203 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:46.048208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:46.048377 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 22:41:46.048530 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 22:41:46.048839 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 22:41:46.051364 357 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.100.188:46107: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a2153cc3e61143849fa047c85042a332"
member_type: NON_VOTER
last_known_addr {
host: "127.23.100.188"
port: 46107
}
attrs {
promote: true
}
}
dest_uuid: "9a9e131374e743a4803b662ca72cf6dd"
cas_config_opid_index: 5
I20260806 22:41:46.051954 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.027743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.23.100.129:34075)
0806 22:41:46.030551 (+ 2808us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:46.030561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:46.030595 (+ 34us) server_negotiation.cc:407] Connection header received
0806 22:41:46.030700 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:46.030709 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:46.031012 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:46.031203 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:46.032857 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.035040 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.037034 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.037999 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.038695 (+ 696us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.044023 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:46.044039 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:46.044059 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:46.044103 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:46.048606 (+ 4503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.049509 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.049555 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.049561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.049744 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.050194 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.050200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.050204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.050572 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:46.050888 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:46.051242 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 22:41:46.051520 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2566,"thread_start_us":2372,"threads_started":1}
I20260806 22:41:46.052505 357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } }
I20260806 22:41:46.054070 645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:46.054390 1484 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a2153cc3e61143849fa047c85042a332 (127.23.100.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.058804 1484 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.100.188:46107
I20260806 22:41:46.059655 1248 master.cc:537] Master@127.23.100.188:46107 shutting down...
I20260806 22:41:46.059528 1518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.059932 1311 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 LEARNER]: Refusing update from remote peer 9a9e131374e743a4803b662ca72cf6dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 22:41:46.059880 1518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:46.060616 1484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 22:41:46.060835 1438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:46.074347 1438 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 FOLLOWER]: Refusing update from remote peer 9a9e131374e743a4803b662ca72cf6dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 22:41:46.080124 1311 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a2153cc3e61143849fa047c85042a332 (127.23.100.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.081305 1484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:46.090912 1352 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: NON_VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: true } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.091251 1352 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: This master's current role is: LEARNER
I20260806 22:41:46.091712 1484 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd: attempting to promote NON_VOTER e04600f32f8c41ed812bcc5bd9332e28 to VOTER
I20260806 22:41:46.092562 1484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } }
I20260806 22:41:46.097330 1311 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 LEARNER]: Refusing update from remote peer 9a9e131374e743a4803b662ca72cf6dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 22:41:46.098055 1522 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 22:41:46.100509 1484 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e04600f32f8c41ed812bcc5bd9332e28 (127.23.100.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.103241 1311 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e04600f32f8c41ed812bcc5bd9332e28 (127.23.100.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.106324 1352 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.106598 1352 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:41:46.108136 1518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.108408 1518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:46.108081 1484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.108640 1484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: This master's current role is: LEADER
W20260806 22:41:46.113868 1405 tablet.cc:2369] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:41:46.116604 1248 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 22:41:46.117003 333 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd -> Peer a2153cc3e61143849fa047c85042a332 (127.23.100.188:46107): Couldn't send request to peer a2153cc3e61143849fa047c85042a332. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 22:41:46.117046 1248 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:46.117264 1248 tablet_replica.cc:330] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: stopping tablet replica
I20260806 22:41:46.134852 1248 master.cc:552] Master@127.23.100.188:46107 shutdown complete.
I20260806 22:41:46.134955 1248 master_runner.cc:305] Clearing existing system tablet
I20260806 22:41:46.135346 1248 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:46.135469 1248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:46.140422 1248 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 22:41:46.142187 1525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:46.142661 1248 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:46.142942 1248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 22:41:46.143057 1248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/wal
uuid: "a2153cc3e61143849fa047c85042a332"
format_stamp: "Formatted at 2026-08-06 22:41:45 on dist-test-slave-zr1t"
server_key: "8b8f71fb3b038e3f5efe8af0f574aa34"
server_key_iv: "5014bd0bee8b1aa8fdd0253d7724f13c"
server_key_version: "encryptionkey@0"
I20260806 22:41:46.148170 1248 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:46.155547 1248 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 22:41:46.155745 1248 log.cc:1196] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 22:41:46.156247 1248 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Deleting consensus metadata
I20260806 22:41:46.156543 1248 master_runner.cc:315] Copying system tablet from 127.23.100.190:45503
I20260806 22:41:46.159300 1248 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: tablet copy: Beginning tablet copy session from remote peer at address 127.23.100.190:45503
I20260806 22:41:46.181398 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.160252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.23.100.190:45503)
0806 22:41:46.160503 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:46.160510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:46.166235 (+ 5725us) server_negotiation.cc:407] Connection header received
0806 22:41:46.166559 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:46.166566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:46.166675 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:46.166796 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:46.167943 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.169104 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.170984 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.171679 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.171833 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.174526 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:46.174535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:46.174540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:46.174582 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:46.178266 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.179126 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.179132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.179136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.179305 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.179722 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.179727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.179730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.180132 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:46.180271 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:46.180802 (+ 531us) server_negotiation.cc:299] Negotiation successful
0806 22:41:46.181043 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260806 22:41:46.181456 1538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.160176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:52754)
0806 22:41:46.166059 (+ 5883us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:46.166084 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:46.166384 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:46.166845 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:46.166851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:46.167461 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:46.167764 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.167777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.169240 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.169246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.170809 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.170826 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.171851 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.171858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.172075 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.172868 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:46.172892 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:46.174278 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:46.178418 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.178425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.178429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.178991 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.179425 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.179430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.179434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.179617 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.180249 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:46.180254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:46.180469 (+ 215us) client_negotiation.cc:769] Sending connection context
0806 22:41:46.180622 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 22:41:46.181001 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5721,"thread_start_us":5565,"threads_started":1}
I20260806 22:41:46.182488 383 tablet_copy_service.cc:140] P 9a9e131374e743a4803b662ca72cf6dd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a2153cc3e61143849fa047c85042a332 ({username='oryx', principal='oryx/127.23.100.188@KRBTEST.COM'} at 127.0.0.1:52754)
I20260806 22:41:46.182678 383 tablet_copy_service.cc:161] P 9a9e131374e743a4803b662ca72cf6dd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a2153cc3e61143849fa047c85042a332 at {username='oryx', principal='oryx/127.23.100.188@KRBTEST.COM'} at 127.0.0.1:52754: session id = a2153cc3e61143849fa047c85042a332-00000000000000000000000000000000
I20260806 22:41:46.186244 383 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 22:41:46.188271 1248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 22:41:46.195550 1248 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: tablet copy: Starting download of 0 data blocks...
I20260806 22:41:46.195919 1248 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: tablet copy: Starting download of 1 WAL segments...
I20260806 22:41:46.198624 1248 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 22:41:46.203680 383 tablet_copy_service.cc:342] P 9a9e131374e743a4803b662ca72cf6dd: Request end of tablet copy session a2153cc3e61143849fa047c85042a332-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.100.188@KRBTEST.COM'} at 127.0.0.1:52754
I20260806 22:41:46.203837 383 tablet_copy_service.cc:433] P 9a9e131374e743a4803b662ca72cf6dd: ending tablet copy session a2153cc3e61143849fa047c85042a332-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a2153cc3e61143849fa047c85042a332
I20260806 22:41:46.207362 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 443
I20260806 22:41:46.226261 1248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:46.231164 1542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:46.231245 1541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:46.238504 1547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:46.245852 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:34075
--local_ip_for_outbound_sockets=127.23.100.129
--tserver_master_addrs=127.23.100.190:45503
--webserver_port=45113
--webserver_interface=127.23.100.129
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab
--principal=oryx/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.100.190:45503,127.23.100.189:37055
--tserver_master_addrs=127.23.100.190:45503,127.23.100.189:37055,127.23.100.188:46107 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:41:46.255191 1248 server_base.cc:1034] running on GCE node
I20260806 22:41:46.255661 1248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:46.256458 1248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:46.257627 1248 hybrid_clock.cc:648] HybridClock initialized: now 1786056106257566 us; error 46 us; skew 500 ppm
I20260806 22:41:46.257937 1248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:46 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056106, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:46.260257 1248 init.cc:377] Logged in from keytab as oryx/127.23.100.188@KRBTEST.COM (short username oryx)
I20260806 22:41:46.265877 1248 webserver.cc:460] Webserver started at http://127.23.100.188:34165/ using document root <none> and password file <none>
I20260806 22:41:46.266330 1248 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:46.266481 1248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:46.270176 1248 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260806 22:41:46.272521 1560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:46.273010 1248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260806 22:41:46.273157 1248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/wal
uuid: "a2153cc3e61143849fa047c85042a332"
format_stamp: "Formatted at 2026-08-06 22:41:45 on dist-test-slave-zr1t"
server_key: "8b8f71fb3b038e3f5efe8af0f574aa34"
server_key_iv: "5014bd0bee8b1aa8fdd0253d7724f13c"
server_key_version: "encryptionkey@0"
I20260806 22:41:46.273389 1248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:46.296046 1248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:46.300256 1248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:46.354293 1248 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.188:46107
I20260806 22:41:46.354357 1642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.188:46107 every 8 connection(s)
I20260806 22:41:46.355451 1248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
I20260806 22:41:46.361837 1647 sys_catalog.cc:263] Verifying existing consensus state
I20260806 22:41:46.366596 1647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Bootstrap starting.
I20260806 22:41:46.387024 1647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 22:41:46.387580 1647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Bootstrap complete.
I20260806 22:41:46.388718 1647 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } }
I20260806 22:41:46.389055 1647 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2153cc3e61143849fa047c85042a332, State: Initialized, Role: LEARNER
I20260806 22:41:46.389323 1647 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } }
I20260806 22:41:46.390699 1647 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:46.390372 1652 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.391021 1652 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: This master's current role is: LEARNER
I20260806 22:41:46.401392 1666 catalog_manager.cc:1259] Loaded cluster ID: 9743b2c273644d7899b4f95bc4699be9
I20260806 22:41:46.401549 1666 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: loading cluster ID for follower catalog manager: success
I20260806 22:41:46.405098 1666 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: acquiring CA information for follower catalog manager: success
I20260806 22:41:46.407572 1666 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 22:41:46.485812 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.477634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46107 (local address 127.0.0.1:36346)
0806 22:41:46.477921 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:46.477940 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:46.478078 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:46.478997 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:46.479004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:46.479048 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:46.479292 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.479310 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.481223 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.481229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.483198 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.483217 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.484912 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.484918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.485141 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.485179 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 22:41:46.485287 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 22:41:46.485450 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260806 22:41:46.486166 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.477887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.23.100.188:46107)
0806 22:41:46.478596 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:46.478602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:46.478620 (+ 18us) server_negotiation.cc:407] Connection header received
0806 22:41:46.478691 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:46.478696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:46.478779 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:46.478904 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:46.479504 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.481054 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.483391 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.484710 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.484926 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.485104 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:46.485341 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 22:41:46.485484 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":411,"threads_started":1}
I20260806 22:41:46.487095 1606 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 22:41:46.487762 1652 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a9e131374e743a4803b662ca72cf6dd. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.488070 1652 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: This master's current role is: LEARNER
W20260806 22:41:46.659169 1551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:46.659456 1551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:46.659554 1551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:46.666950 1551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:46.667062 1551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:46.667179 1551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:41:46.677312 1551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab
--principal=oryx/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:34075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=45113
--tserver_master_addrs=127.23.100.190:45503,127.23.100.189:37055,127.23.100.188:46107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:46.678642 1551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:46.679883 1551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:46.693606 1681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:46.693647 1683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:46.694355 1679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:46.699067 1551 server_base.cc:1034] running on GCE node
I20260806 22:41:46.699754 1551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:46.700706 1551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:46.701936 1551 hybrid_clock.cc:648] HybridClock initialized: now 1786056106701871 us; error 51 us; skew 500 ppm
I20260806 22:41:46.702253 1551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:46 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056106, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:46.705453 1551 init.cc:377] Logged in from keytab as oryx/127.23.100.129@KRBTEST.COM (short username oryx)
I20260806 22:41:46.707010 1551 webserver.cc:460] Webserver started at http://127.23.100.129:45113/ using document root <none> and password file <none>
I20260806 22:41:46.707710 1551 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:46.707877 1551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:46.713311 1551 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:41:46.716065 1693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:46.717478 1551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:41:46.717721 1551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "cd37baa871ee4ebbb82552241b0069b2"
format_stamp: "Formatted at 2026-08-06 22:41:43 on dist-test-slave-zr1t"
server_key: "b3a493a49962ffd3a12a201ad66ebebe"
server_key_iv: "b317dca1ff3a371fd5ed589d3ae5b5aa"
server_key_version: "encryptionkey@0"
I20260806 22:41:46.718708 1551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:46.738518 1551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:46.742951 1551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:46.743255 1551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:46.744328 1551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:46.746023 1551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:46.746146 1551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:46.746271 1551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:46.746361 1551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:46.781661 1551 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:34075
I20260806 22:41:46.781728 1868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:34075 every 8 connection(s)
I20260806 22:41:46.783219 1551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:41:46.787063 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 1551
I20260806 22:41:46.787420 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 661
Aug 06 22:41:46 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056106, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
Aug 06 22:41:46 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056106, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.189@KRBTEST.COM
Aug 06 22:41:46 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.130@KRBTEST.COM
Aug 06 22:41:46 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056106, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.188@KRBTEST.COM
I20260806 22:41:46.811079 1877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.786158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:48711 (local address 127.23.100.190:45503)
0806 22:41:46.786651 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:46.786659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:46.790754 (+ 4095us) server_negotiation.cc:407] Connection header received
0806 22:41:46.790839 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:46.790847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:46.790943 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:46.791066 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:46.793485 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.794588 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.797321 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.798060 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.798205 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.802995 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:46.803003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:46.803008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:46.803056 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:46.807194 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.808181 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.808189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.808196 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.808358 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.808959 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.808965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.808968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.809252 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:46.809408 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:46.810419 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0806 22:41:46.810692 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":214,"threads_started":1}
I20260806 22:41:46.811940 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.789480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:49781 (local address 127.23.100.188:46107)
0806 22:41:46.789681 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:46.789687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:46.791873 (+ 2186us) server_negotiation.cc:407] Connection header received
0806 22:41:46.792043 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:46.792049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:46.792137 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:46.792230 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:46.793794 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.794913 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.797844 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.798484 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.798581 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.804324 (+ 5743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:46.804333 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:46.804338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:46.804399 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:46.808413 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.809564 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.809570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.809573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.809736 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.810221 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.810226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.810230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.810601 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:46.810744 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:46.811322 (+ 578us) server_negotiation.cc:299] Negotiation successful
0806 22:41:46.811590 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 22:41:46.812752 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.787864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:51235 (local address 127.23.100.189:37055)
0806 22:41:46.792356 (+ 4492us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:46.792364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:46.792389 (+ 25us) server_negotiation.cc:407] Connection header received
0806 22:41:46.792460 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:46.792466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:46.792565 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:46.794021 (+ 1456us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:46.794207 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.795263 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.797359 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.798002 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.800945 (+ 2943us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.804270 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:46.804276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:46.804280 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:46.804325 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:46.808143 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.809489 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.809494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.809497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.809664 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.810203 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.810209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.810215 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.810573 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:46.812026 (+ 1453us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:46.812169 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 22:41:46.812397 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4363,"thread_start_us":4457,"threads_started":1}
I20260806 22:41:46.813054 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.787406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:37055 (local address 127.23.100.129:51235)
0806 22:41:46.788093 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:46.788121 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:46.790397 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:46.792775 (+ 2378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:46.792785 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:46.793421 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:46.793820 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.793831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.795425 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.795431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.797179 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.797195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.798237 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.798244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.798553 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.799483 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:46.799523 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:46.804020 (+ 4497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:46.808283 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.808289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.808292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.808889 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.809801 (+ 912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.809805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.809808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.810012 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.810719 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:46.810723 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:46.810867 (+ 144us) client_negotiation.cc:769] Sending connection context
0806 22:41:46.811035 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 22:41:46.811591 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":5,"thread_start_us":661,"threads_started":1}
I20260806 22:41:46.813148 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.789045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46107 (local address 127.23.100.129:49781)
0806 22:41:46.791749 (+ 2704us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:46.791792 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:46.791929 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:46.792479 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:46.792485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:46.793198 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:46.793595 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.793607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.795107 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.795112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.797681 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.797696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.798926 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.798933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.799139 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.799752 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:46.799773 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:46.804086 (+ 4313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:46.808594 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.808598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.808601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.809388 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.809906 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.809911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.809914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.810092 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.810833 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:46.810837 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:46.811005 (+ 168us) client_negotiation.cc:769] Sending connection context
0806 22:41:46.812203 (+ 1198us) client_negotiation.cc:240] Negotiation successful
0806 22:41:46.812510 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2544,"mutex_wait_us":729,"thread_start_us":2383,"threads_started":1}
I20260806 22:41:46.813544 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.786716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.129:48711)
0806 22:41:46.787594 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:46.787659 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:46.790382 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:46.791369 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:46.791385 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:46.792360 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:46.793264 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.793291 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.794807 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.794812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.797170 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:46.797182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.798468 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.798477 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:46.798676 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.799394 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:46.799445 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:46.802653 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:46.807372 (+ 4719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.807382 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.807399 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.808025 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.808524 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:46.808528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:46.808532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:46.808817 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:46.809411 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:46.809421 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:46.809960 (+ 539us) client_negotiation.cc:769] Sending connection context
0806 22:41:46.811349 (+ 1389us) client_negotiation.cc:240] Negotiation successful
0806 22:41:46.811717 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":135,"thread_start_us":2906,"threads_started":1}
I20260806 22:41:46.814159 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:46.788959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.23.100.130:46003)
0806 22:41:46.789625 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:46.789632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:46.789655 (+ 23us) server_negotiation.cc:407] Connection header received
0806 22:41:46.789711 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:46.789716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:46.789899 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:46.790064 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:46.792224 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.793775 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.795805 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:46.796643 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:46.796778 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:46.804145 (+ 7367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:46.804166 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:46.804185 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:46.804242 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:46.808601 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.809591 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.809600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.809603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.809761 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:46.810221 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:46.810226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:46.810230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:46.810637 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:46.810940 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:46.812763 (+ 1823us) server_negotiation.cc:299] Negotiation successful
0806 22:41:46.813732 (+ 969us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":401,"threads_started":1}
I20260806 22:41:46.814985 1871 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45503
I20260806 22:41:46.815018 1872 heartbeater.cc:347] Connected to a master server at 127.23.100.189:37055
I20260806 22:41:46.815289 1872 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:46.815299 1871 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:46.815446 859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:46.816306 1871 heartbeater.cc:511] Master 127.23.100.190:45503 requested a full tablet report, sending...
I20260806 22:41:46.816357 1869 heartbeater.cc:347] Connected to a master server at 127.23.100.188:46107
I20260806 22:41:46.816331 1872 heartbeater.cc:511] Master 127.23.100.189:37055 requested a full tablet report, sending...
I20260806 22:41:46.816519 1869 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:46.816987 1869 heartbeater.cc:511] Master 127.23.100.188:46107 requested a full tablet report, sending...
I20260806 22:41:46.818881 1296 ts_manager.cc:194] Registered new tserver with Master: cd37baa871ee4ebbb82552241b0069b2 (127.23.100.129:34075)
I20260806 22:41:46.819327 1587 ts_manager.cc:194] Registered new tserver with Master: cd37baa871ee4ebbb82552241b0069b2 (127.23.100.129:34075)
I20260806 22:41:46.819870 357 ts_manager.cc:194] Re-registered known tserver with Master: cd37baa871ee4ebbb82552241b0069b2 (127.23.100.129:34075)
I20260806 22:41:46.820901 357 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:48711
I20260806 22:41:46.919968 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 661
I20260806 22:41:46.946889 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:46003
--local_ip_for_outbound_sockets=127.23.100.130
--tserver_master_addrs=127.23.100.190:45503
--webserver_port=42979
--webserver_interface=127.23.100.130
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab
--principal=oryx/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.100.190:45503,127.23.100.189:37055
--tserver_master_addrs=127.23.100.190:45503,127.23.100.189:37055,127.23.100.188:46107 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:41:46.964720 1484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: NON_VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: true } } }
I20260806 22:41:46.965173 1484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:46.966594 1484 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd: attempting to promote NON_VOTER a2153cc3e61143849fa047c85042a332 to VOTER
I20260806 22:41:46.967202 1484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } }
I20260806 22:41:46.969122 1311 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 FOLLOWER]: Refusing update from remote peer 9a9e131374e743a4803b662ca72cf6dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 22:41:46.969806 1606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 LEARNER]: Refusing update from remote peer 9a9e131374e743a4803b662ca72cf6dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 22:41:46.969851 1892 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 22:41:46.970431 1484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 22:41:46.973021 1484 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2153cc3e61143849fa047c85042a332 (127.23.100.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:46.973637 1311 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2153cc3e61143849fa047c85042a332 (127.23.100.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:46.977412 1893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:46.977741 1893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:46.978039 1893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:46.978240 1893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:46.982017 1897 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:46.982440 1897 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:41:46.984251 1892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:46.984563 1892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:46.984928 1606 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2153cc3e61143849fa047c85042a332 (127.23.100.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:46.989413 1898 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a9e131374e743a4803b662ca72cf6dd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:46.989758 1898 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 22:41:47.295986 1891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:47.296303 1891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:47.296478 1891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:47.304124 1891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:47.304291 1891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:47.304472 1891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:41:47.314851 1891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab
--principal=oryx/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:46003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=42979
--tserver_master_addrs=127.23.100.190:45503,127.23.100.189:37055,127.23.100.188:46107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:47.316628 1891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:47.318079 1891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:47.331674 1907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:47.331588 1912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:47.331633 1908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:47.333045 1891 server_base.cc:1034] running on GCE node
I20260806 22:41:47.333752 1891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:47.334648 1891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:47.335861 1891 hybrid_clock.cc:648] HybridClock initialized: now 1786056107335800 us; error 50 us; skew 500 ppm
I20260806 22:41:47.336221 1891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:47 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056107, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:47.339425 1891 init.cc:377] Logged in from keytab as oryx/127.23.100.130@KRBTEST.COM (short username oryx)
I20260806 22:41:47.341035 1891 webserver.cc:460] Webserver started at http://127.23.100.130:42979/ using document root <none> and password file <none>
I20260806 22:41:47.341854 1891 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:47.342038 1891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:47.347878 1891 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 22:41:47.350689 1924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:47.352084 1891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:41:47.352257 1891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "0a62863097244864bed64a9809dfdcde"
format_stamp: "Formatted at 2026-08-06 22:41:43 on dist-test-slave-zr1t"
server_key: "486382038629eae31ee863011d6cf5df"
server_key_iv: "15fe9b747f9b05f48851ce7e33961770"
server_key_version: "encryptionkey@0"
I20260806 22:41:47.353246 1891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:47.366622 1891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:47.371102 1891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:47.371416 1891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:47.372404 1891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:47.373849 1891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:47.373979 1891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:47.374096 1891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:47.374155 1891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:47.408682 1891 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:46003
I20260806 22:41:47.408788 2087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:46003 every 8 connection(s)
I20260806 22:41:47.410313 1891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:41:47.415112 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 1891
I20260806 22:41:47.415452 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 875
Aug 06 22:41:47 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056107, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
Aug 06 22:41:47 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056107, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.189@KRBTEST.COM
Aug 06 22:41:47 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.131@KRBTEST.COM
I20260806 22:41:47.452554 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:47.414185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:33169 (local address 127.23.100.190:45503)
0806 22:41:47.415248 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:47.415255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:47.432268 (+ 17013us) server_negotiation.cc:407] Connection header received
0806 22:41:47.432571 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:47.432578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:47.432671 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:47.432857 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:47.434793 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.435910 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.438573 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.439221 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.439335 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:47.444203 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:47.444208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:47.444211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:47.444282 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:47.449178 (+ 4896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:47.450327 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:47.450332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:47.450335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:47.450494 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:47.450896 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:47.450903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:47.450909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:47.451284 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:47.451448 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:47.451900 (+ 452us) server_negotiation.cc:299] Negotiation successful
0806 22:41:47.452151 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":821,"threads_started":1}
I20260806 22:41:47.452692 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:47.413744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:51771 (local address 127.23.100.189:37055)
0806 22:41:47.414348 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:47.414357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:47.427624 (+ 13267us) server_negotiation.cc:407] Connection header received
0806 22:41:47.431672 (+ 4048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:47.431682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:47.431814 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:47.431934 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:47.434866 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.436341 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.438918 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.439736 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.439864 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:47.444642 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:47.444649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:47.444654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:47.444709 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:47.448775 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:47.449789 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:47.449795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:47.449799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:47.449949 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:47.450498 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:47.450503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:47.450506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:47.450929 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:47.451099 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:47.452104 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0806 22:41:47.452343 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":586,"threads_started":1}
I20260806 22:41:47.453902 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:47.416660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.23.100.131:34259)
0806 22:41:47.428248 (+ 11588us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:47.428261 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:47.428863 (+ 602us) server_negotiation.cc:407] Connection header received
0806 22:41:47.429434 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:47.429448 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:47.429758 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:47.429901 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:47.432611 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.434326 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.436467 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.437297 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.437398 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:47.445855 (+ 8457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:47.445872 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:47.445890 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:47.445940 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:47.449891 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:47.450798 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:47.450808 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:47.450812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:47.450986 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:47.451443 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:47.451458 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:47.451463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:47.451876 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:47.452157 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:47.453195 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0806 22:41:47.453472 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11351,"thread_start_us":11582,"threads_started":1}
I20260806 22:41:47.453905 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:47.414727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.130:33169)
0806 22:41:47.432091 (+ 17364us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:47.432113 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:47.432390 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:47.433095 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:47.433100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:47.433730 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:47.434591 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:47.434619 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.436134 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.436142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:47.438359 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:47.438385 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.439716 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.439725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:47.440043 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:47.440835 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:47.440878 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:47.443953 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:47.449616 (+ 5663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:47.449620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:47.449624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:47.450211 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:47.450616 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:47.450621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:47.450624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:47.450774 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:47.451422 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:47.451427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:47.451559 (+ 132us) client_negotiation.cc:769] Sending connection context
0806 22:41:47.451793 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 22:41:47.453171 (+ 1378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17223,"mutex_wait_us":984,"thread_start_us":12000,"threads_started":1}
I20260806 22:41:47.453926 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:47.413653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:37055 (local address 127.23.100.130:51771)
0806 22:41:47.427337 (+ 13684us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:47.427417 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:47.431234 (+ 3817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:47.432420 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:47.432433 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:47.433442 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:47.434593 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:47.434622 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.436595 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.436603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:47.438666 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:47.438685 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.440207 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.440216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:47.440457 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:47.441189 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:47.441220 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:47.444415 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:47.448991 (+ 4576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:47.449000 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:47.449017 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:47.449608 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:47.450087 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:47.450096 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:47.450100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:47.450371 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:47.451071 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:47.451079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:47.451478 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 22:41:47.451793 (+ 315us) client_negotiation.cc:240] Negotiation successful
0806 22:41:47.452347 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13237,"mutex_wait_us":126,"thread_start_us":18018,"threads_started":1}
I20260806 22:41:47.455543 2093 heartbeater.cc:347] Connected to a master server at 127.23.100.189:37055
I20260806 22:41:47.455797 2093 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:47.456317 2090 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45503
I20260806 22:41:47.456548 2090 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:47.456632 1069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:47.456698 2093 heartbeater.cc:511] Master 127.23.100.189:37055 requested a full tablet report, sending...
I20260806 22:41:47.457149 2090 heartbeater.cc:511] Master 127.23.100.190:45503 requested a full tablet report, sending...
I20260806 22:41:47.458542 357 ts_manager.cc:194] Re-registered known tserver with Master: 0a62863097244864bed64a9809dfdcde (127.23.100.130:46003)
Aug 06 22:41:47 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056107, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.188@KRBTEST.COM
I20260806 22:41:47.459791 1296 ts_manager.cc:194] Registered new tserver with Master: 0a62863097244864bed64a9809dfdcde (127.23.100.130:46003)
I20260806 22:41:47.460048 357 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:33169
I20260806 22:41:47.467849 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:47.438495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46107 (local address 127.23.100.130:33567)
0806 22:41:47.446025 (+ 7530us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:47.446043 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:47.446266 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:47.446971 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:47.446976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:47.447606 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:47.447951 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:47.447965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.449584 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.449590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:47.453379 (+ 3789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:47.453396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.454938 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.454945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:47.455145 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:47.457878 (+ 2733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:47.457894 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:47.460724 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:47.464908 (+ 4184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:47.464915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:47.464919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:47.465427 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:47.465860 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:47.465865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:47.465868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:47.466045 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:47.466698 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:47.466703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:47.466875 (+ 172us) client_negotiation.cc:769] Sending connection context
0806 22:41:47.467041 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 22:41:47.467382 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7393,"thread_start_us":7213,"threads_started":1}
I20260806 22:41:47.468699 2107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:47.432565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:33567 (local address 127.23.100.188:46107)
0806 22:41:47.443139 (+ 10574us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:47.443143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:47.446386 (+ 3243us) server_negotiation.cc:407] Connection header received
0806 22:41:47.446461 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:47.446466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:47.446631 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:47.446821 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:47.448162 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.449399 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.453580 (+ 4181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:47.454325 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:47.454444 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:47.460963 (+ 6519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:47.460973 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:47.460981 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:47.461654 (+ 673us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:47.464772 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:47.465593 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:47.465598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:47.465600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:47.465717 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:47.466172 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:47.466180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:47.466185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:47.466547 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:47.466718 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:47.468158 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0806 22:41:47.468428 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10455,"thread_start_us":11707,"threads_started":1}
I20260806 22:41:47.469626 2088 heartbeater.cc:347] Connected to a master server at 127.23.100.188:46107
I20260806 22:41:47.469782 2088 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:47.470170 2088 heartbeater.cc:511] Master 127.23.100.188:46107 requested a full tablet report, sending...
I20260806 22:41:47.471159 1587 ts_manager.cc:194] Registered new tserver with Master: 0a62863097244864bed64a9809dfdcde (127.23.100.130:46003)
I20260806 22:41:47.550016 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 875
I20260806 22:41:47.575287 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:34259
--local_ip_for_outbound_sockets=127.23.100.131
--tserver_master_addrs=127.23.100.190:45503
--webserver_port=44849
--webserver_interface=127.23.100.131
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab
--principal=oryx/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.100.190:45503,127.23.100.189:37055
--tserver_master_addrs=127.23.100.190:45503,127.23.100.189:37055,127.23.100.188:46107 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:41:47.824729 1871 heartbeater.cc:503] Master 127.23.100.190:45503 was elected leader, sending a full tablet report...
W20260806 22:41:47.926156 2111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:47.926497 2111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:47.926662 2111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:47.934221 2111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:47.934388 2111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:47.934564 2111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:41:47.945390 2111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab
--principal=oryx/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:34259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=44849
--tserver_master_addrs=127.23.100.190:45503,127.23.100.189:37055,127.23.100.188:46107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:47.947021 2111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:47.948446 2111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:47.963271 2117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:47.963394 2116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:47.963420 2121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:47.964056 2111 server_base.cc:1034] running on GCE node
I20260806 22:41:47.964716 2111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:47.965790 2111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:47.967012 2111 hybrid_clock.cc:648] HybridClock initialized: now 1786056107966946 us; error 55 us; skew 500 ppm
I20260806 22:41:47.967356 2111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:47 dist-test-slave-zr1t krb5kdc[32760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056107, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:47.970841 2111 init.cc:377] Logged in from keytab as oryx/127.23.100.131@KRBTEST.COM (short username oryx)
I20260806 22:41:47.972394 2111 webserver.cc:460] Webserver started at http://127.23.100.131:44849/ using document root <none> and password file <none>
I20260806 22:41:47.973088 2111 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:47.973208 2111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:47.978751 2111 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:41:47.981444 2131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:47.982779 2111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 22:41:47.982954 2111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "9517100acaf74cc3a6b7caf88c305b03"
format_stamp: "Formatted at 2026-08-06 22:41:44 on dist-test-slave-zr1t"
server_key: "dd18004417caaf023d1260f54a79405f"
server_key_iv: "efc353fd800452c1622853c6f08ff94e"
server_key_version: "encryptionkey@0"
I20260806 22:41:47.983952 2111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:48.004185 2111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:48.009042 2111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:48.009351 2111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:48.010416 2111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:41:48.011819 2111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:48.011937 2111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:48.012090 2111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:48.012171 2111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:48.048578 2111 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:34259
I20260806 22:41:48.048663 2316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:34259 every 8 connection(s)
I20260806 22:41:48.050144 2111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:41:48.052985 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 2111
Aug 06 22:41:48 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056107, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
Aug 06 22:41:48 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056107, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.189@KRBTEST.COM
I20260806 22:41:48.089913 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.053242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:39859 (local address 127.23.100.190:45503)
0806 22:41:48.053778 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.053786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.066771 (+ 12985us) server_negotiation.cc:407] Connection header received
0806 22:41:48.068964 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.068970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.069075 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.069203 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:48.072844 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.074078 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.076131 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.076844 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.076980 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.081509 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:48.081538 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:48.081542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:48.081583 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:48.085901 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.086917 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.086925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.086932 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.087104 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.087622 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.087630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.087636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.088050 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:48.088215 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.089114 (+ 899us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.089409 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":296,"threads_started":1}
I20260806 22:41:48.091190 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.053831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.131:39859)
0806 22:41:48.066230 (+ 12399us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:48.066283 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:48.068671 (+ 2388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:48.070349 (+ 1678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:48.070362 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:48.071428 (+ 1066us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:48.072605 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.072634 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.074532 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.074536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.075975 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.075989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.077061 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.077069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.077300 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.078161 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:48.078193 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:48.081233 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:48.086041 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:48.086053 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:48.086072 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:48.086793 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:48.087216 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:48.087220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:48.087223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:48.087507 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:48.088175 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:48.088182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:48.088644 (+ 462us) client_negotiation.cc:769] Sending connection context
0806 22:41:48.088979 (+ 335us) client_negotiation.cc:240] Negotiation successful
0806 22:41:48.089595 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12083,"mutex_wait_us":181,"spinlock_wait_cycles":20352,"thread_start_us":11833,"threads_started":1}
I20260806 22:41:48.092782 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.054182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:45809 (local address 127.23.100.189:37055)
0806 22:41:48.065463 (+ 11281us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.065472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.069573 (+ 4101us) server_negotiation.cc:407] Connection header received
0806 22:41:48.069762 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.069769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.069876 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.069998 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:48.073911 (+ 3913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.075428 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.078429 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.079067 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.079198 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.084455 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:48.084464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:48.084472 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:48.084528 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:48.089020 (+ 4492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.089904 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.089910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.089914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.090075 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.090520 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.090526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.090529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.090956 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:48.091142 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.091715 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.092187 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11080,"thread_start_us":11286,"threads_started":1}
I20260806 22:41:48.094596 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.054103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:37055 (local address 127.23.100.131:45809)
0806 22:41:48.069442 (+ 15339us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:48.069458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:48.069622 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:48.070349 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:48.070362 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:48.071428 (+ 1066us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:48.072605 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.072634 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.075761 (+ 3127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.075768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.078276 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.078292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.080180 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.080193 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.080360 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.081056 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:48.081084 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:48.084193 (+ 3109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:48.089161 (+ 4968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:48.089166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:48.089169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:48.089749 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:48.090233 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:48.090237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:48.090241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:48.090387 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:48.091235 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:48.091240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:48.091399 (+ 159us) client_negotiation.cc:769] Sending connection context
0806 22:41:48.093624 (+ 2225us) client_negotiation.cc:240] Negotiation successful
0806 22:41:48.093973 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15213,"mutex_wait_us":98,"spinlock_wait_cycles":11136,"thread_start_us":15059,"threads_started":1}
I20260806 22:41:48.094655 2319 heartbeater.cc:347] Connected to a master server at 127.23.100.190:45503
I20260806 22:41:48.095077 2319 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:48.096088 2319 heartbeater.cc:511] Master 127.23.100.190:45503 requested a full tablet report, sending...
I20260806 22:41:48.098065 357 ts_manager.cc:194] Re-registered known tserver with Master: 9517100acaf74cc3a6b7caf88c305b03 (127.23.100.131:34259)
I20260806 22:41:48.099352 357 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:39859
Aug 06 22:41:48 dist-test-slave-zr1t krb5kdc[32760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056107, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.188@KRBTEST.COM
I20260806 22:41:48.113355 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.080405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.23.100.189:37055)
0806 22:41:48.093182 (+ 12777us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.093191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.093214 (+ 23us) server_negotiation.cc:407] Connection header received
0806 22:41:48.093277 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.093285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.093382 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.093487 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:48.095741 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.097226 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.100148 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.101114 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.101247 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.105434 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:48.105440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:48.105445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:48.105485 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:48.109053 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.110038 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.110044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.110047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.110209 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.110648 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.110652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.110656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.111054 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:48.111317 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.112733 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.112984 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12543,"threads_started":1}
I20260806 22:41:48.115130 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.069971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:53509 (local address 127.23.100.188:46107)
0806 22:41:48.073055 (+ 3084us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.073061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.073852 (+ 791us) server_negotiation.cc:407] Connection header received
0806 22:41:48.074097 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.074104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.074190 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.074272 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:48.079718 (+ 5446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.081251 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.091024 (+ 9773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.091815 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.091932 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.107287 (+ 15355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:48.107297 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:48.107304 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:48.107359 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:48.111430 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.112411 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.112419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.112425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.112596 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.113087 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.113095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.113101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.113556 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:48.113707 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.114335 (+ 628us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.114630 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2959,"thread_start_us":2824,"threads_started":1}
I20260806 22:41:48.116653 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.073081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46107 (local address 127.23.100.131:53509)
0806 22:41:48.073752 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:48.073770 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:48.073958 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:48.076231 (+ 2273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:48.076235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:48.076856 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:48.079500 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.079513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.086093 (+ 6580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.086102 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.090818 (+ 4716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.090838 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.102881 (+ 12043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.102889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.103129 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.103765 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:48.103794 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:48.107008 (+ 3214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:48.111662 (+ 4654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:48.111669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:48.111676 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:48.112244 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:48.112757 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:48.112765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:48.112771 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:48.112952 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:48.113800 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:48.113806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:48.114006 (+ 200us) client_negotiation.cc:769] Sending connection context
0806 22:41:48.115647 (+ 1641us) client_negotiation.cc:240] Negotiation successful
0806 22:41:48.116091 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":388,"threads_started":1}
I20260806 22:41:48.118264 2318 heartbeater.cc:347] Connected to a master server at 127.23.100.188:46107
I20260806 22:41:48.118481 2318 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:48.119031 2318 heartbeater.cc:511] Master 127.23.100.188:46107 requested a full tablet report, sending...
I20260806 22:41:48.120698 1587 ts_manager.cc:194] Registered new tserver with Master: 9517100acaf74cc3a6b7caf88c305b03 (127.23.100.131:34259)
I20260806 22:41:48.122596 2320 heartbeater.cc:347] Connected to a master server at 127.23.100.189:37055
I20260806 22:41:48.122762 2320 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:48.123148 2320 heartbeater.cc:511] Master 127.23.100.189:37055 requested a full tablet report, sending...
I20260806 22:41:48.124248 1296 ts_manager.cc:194] Registered new tserver with Master: 9517100acaf74cc3a6b7caf88c305b03 (127.23.100.131:34259)
I20260806 22:41:48.124500 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.081741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.23.100.190:45503)
0806 22:41:48.090553 (+ 8812us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.090560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.105973 (+ 15413us) server_negotiation.cc:407] Connection header received
0806 22:41:48.107199 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.107207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.107311 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.107406 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:48.109647 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.110752 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.112683 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.113306 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.113428 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.117428 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:48.117437 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:48.117441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:48.117500 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:48.121204 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.122125 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.122134 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.122140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.122325 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.122823 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.122829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.122834 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.123216 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:48.123560 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.123854 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.124090 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8686,"thread_start_us":34795,"threads_started":1}
I20260806 22:41:48.131349 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.083323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.23.100.188:46107)
0806 22:41:48.114616 (+ 31293us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.114622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.114642 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:41:48.114694 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.114698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.114773 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.114888 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:48.117592 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.118940 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.121366 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.122094 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.122211 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.125494 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:48.125500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:48.125504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:48.125565 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:48.128607 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.129215 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.129222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.129225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.129389 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.129794 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.129798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.129802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.130184 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:48.130422 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.130727 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.130962 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31146,"thread_start_us":30973,"threads_started":1}
I20260806 22:41:48.129191 357 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:48.137933 357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:48.173365 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.159813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:34075 (local address 127.0.0.1:44958)
0806 22:41:48.161138 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:48.161157 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:48.161300 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:48.162547 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:48.162554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:48.162589 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:48.163058 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.163074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.166511 (+ 3437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.166520 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.169539 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.169562 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.172549 (+ 2987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.172555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.172781 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.172825 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 22:41:48.172928 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 22:41:48.173083 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"thread_start_us":1029,"threads_started":1}
I20260806 22:41:48.174611 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.159970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.23.100.129:34075)
0806 22:41:48.160954 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.160963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.161464 (+ 501us) server_negotiation.cc:407] Connection header received
0806 22:41:48.161640 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.161653 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.162014 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.162280 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:48.163302 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.166287 (+ 2985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.169794 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.171941 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.172173 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.172384 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.174090 (+ 1706us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.174280 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":343,"threads_started":1}
I20260806 22:41:48.179981 1767 tablet_service.cc:1467] Processing CreateTablet for tablet 7808e1cf4384423981daacea7ae7af3a (DEFAULT_TABLE table=test-table [id=a4ee016c79c6421f9337d44cff4ff312]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:48.181569 1767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7808e1cf4384423981daacea7ae7af3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:48.181753 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.161271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:34259 (local address 127.0.0.1:38660)
0806 22:41:48.173734 (+ 12463us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:48.173752 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:48.173859 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:48.174789 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:48.174795 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:48.174822 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:48.175006 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.175017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.177190 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.177196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.178755 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.178776 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.180885 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.180891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.181033 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.181057 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 22:41:48.181166 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 22:41:48.181339 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12373,"mutex_wait_us":1,"thread_start_us":14477,"threads_started":1}
I20260806 22:41:48.182070 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.162074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.23.100.131:34259)
0806 22:41:48.172324 (+ 10250us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.172330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.173838 (+ 1508us) server_negotiation.cc:407] Connection header received
0806 22:41:48.174081 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.174089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.174418 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.174651 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:48.175536 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.177056 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.178929 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.180203 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.180369 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.180563 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.181225 (+ 662us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.181404 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10093,"thread_start_us":9926,"threads_started":1}
I20260806 22:41:48.184347 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.161139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:46003 (local address 127.0.0.1:39636)
0806 22:41:48.172560 (+ 11421us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:48.172579 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:48.172729 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:48.175126 (+ 2397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:48.175130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:48.177216 (+ 2086us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:48.177701 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.177714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.179680 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.179686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.181832 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.181848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.183775 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.183780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.183905 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.183928 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 22:41:48.183994 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 22:41:48.184086 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11268,"mutex_wait_us":30,"thread_start_us":11686,"threads_started":1}
I20260806 22:41:48.184530 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.161678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.23.100.130:46003)
0806 22:41:48.171753 (+ 10075us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.171785 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.172717 (+ 932us) server_negotiation.cc:407] Connection header received
0806 22:41:48.172904 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.172917 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.174866 (+ 1949us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.175137 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:48.177938 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.179456 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.182043 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.183574 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.183772 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.183934 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.184058 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.184236 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9899,"thread_start_us":9790,"threads_started":1}
I20260806 22:41:48.188192 1997 tablet_service.cc:1467] Processing CreateTablet for tablet 7808e1cf4384423981daacea7ae7af3a (DEFAULT_TABLE table=test-table [id=a4ee016c79c6421f9337d44cff4ff312]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:48.189493 1997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7808e1cf4384423981daacea7ae7af3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:48.189121 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 7808e1cf4384423981daacea7ae7af3a (DEFAULT_TABLE table=test-table [id=a4ee016c79c6421f9337d44cff4ff312]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:48.190574 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7808e1cf4384423981daacea7ae7af3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:48.198469 2368 tablet_bootstrap.cc:492] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: Bootstrap starting.
I20260806 22:41:48.202316 2368 tablet_bootstrap.cc:654] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:48.203037 2369 tablet_bootstrap.cc:492] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde: Bootstrap starting.
I20260806 22:41:48.203930 2368 log.cc:824] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:48.207056 2369 tablet_bootstrap.cc:654] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:48.207104 2368 tablet_bootstrap.cc:492] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: No bootstrap required, opened a new log
I20260806 22:41:48.207404 2368 ts_tablet_manager.cc:1397] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260806 22:41:48.208321 2369 log.cc:824] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:48.210834 2369 tablet_bootstrap.cc:492] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde: No bootstrap required, opened a new log
I20260806 22:41:48.211066 2371 tablet_bootstrap.cc:492] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: Bootstrap starting.
I20260806 22:41:48.211117 2369 ts_tablet_manager.cc:1397] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260806 22:41:48.214716 2371 tablet_bootstrap.cc:654] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:48.216051 2371 log.cc:824] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:48.216955 2368 raft_consensus.cc:348] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:48.217453 2368 raft_consensus.cc:374] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:48.217598 2368 raft_consensus.cc:729] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37baa871ee4ebbb82552241b0069b2, State: Initialized, Role: FOLLOWER
I20260806 22:41:48.218212 2368 consensus_queue.cc:260] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:48.219578 2371 tablet_bootstrap.cc:492] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: No bootstrap required, opened a new log
I20260806 22:41:48.219858 2371 ts_tablet_manager.cc:1397] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260806 22:41:48.219982 2368 ts_tablet_manager.cc:1428] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260806 22:41:48.222312 2369 raft_consensus.cc:348] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:48.223284 2369 raft_consensus.cc:374] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:48.223457 2369 raft_consensus.cc:729] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a62863097244864bed64a9809dfdcde, State: Initialized, Role: FOLLOWER
I20260806 22:41:48.224246 2369 consensus_queue.cc:260] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:48.227952 2369 ts_tablet_manager.cc:1428] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde: Time spent starting tablet: real 0.017s user 0.011s sys 0.002s
I20260806 22:41:48.228273 2090 heartbeater.cc:503] Master 127.23.100.190:45503 was elected leader, sending a full tablet report...
I20260806 22:41:48.231096 2371 raft_consensus.cc:348] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:48.231572 2371 raft_consensus.cc:374] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:48.231681 2371 raft_consensus.cc:729] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9517100acaf74cc3a6b7caf88c305b03, State: Initialized, Role: FOLLOWER
I20260806 22:41:48.232416 2371 consensus_queue.cc:260] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:48.233422 2319 heartbeater.cc:503] Master 127.23.100.190:45503 was elected leader, sending a full tablet report...
I20260806 22:41:48.234122 2371 ts_tablet_manager.cc:1428] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260806 22:41:48.286612 1873 tablet.cc:2369] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:41:48.298168 2376 raft_consensus.cc:482] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:48.298483 2376 raft_consensus.cc:504] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:48.300086 2376 leader_election.cc:288] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd37baa871ee4ebbb82552241b0069b2 (127.23.100.129:34075), 9517100acaf74cc3a6b7caf88c305b03 (127.23.100.131:34259)
W20260806 22:41:48.303280 2321 tablet.cc:2369] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:41:48.308087 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.301473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:54373 (local address 127.23.100.131:34259)
0806 22:41:48.301868 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.301875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.302012 (+ 137us) server_negotiation.cc:407] Connection header received
0806 22:41:48.302176 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.302180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.302273 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.302372 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:48.303134 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.304313 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.306314 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.307079 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.307276 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.307351 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.307704 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.307834 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260806 22:41:48.308264 2383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.301203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:34259 (local address 127.23.100.130:54373)
0806 22:41:48.301882 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:48.301901 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:48.302040 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:48.302448 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:48.302452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:48.302483 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:48.302880 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.302890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.304469 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.304474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.306148 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.306160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.307295 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.307301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.307532 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.307565 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 22:41:48.307675 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 22:41:48.307814 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":30,"thread_start_us":342,"threads_started":1}
I20260806 22:41:48.308297 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.300617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:41487 (local address 127.23.100.129:34075)
0806 22:41:48.300805 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.300809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.301309 (+ 500us) server_negotiation.cc:407] Connection header received
0806 22:41:48.301790 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.301796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.301884 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.301992 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:48.302943 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.304069 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.306277 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.307167 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.307379 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.307473 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.307847 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.307998 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 22:41:48.308758 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.300463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:34075 (local address 127.23.100.130:41487)
0806 22:41:48.301158 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:48.301178 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:48.301393 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:48.302239 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:48.302260 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:48.302285 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:48.302752 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.302765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.304246 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.304253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.306068 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:48.306084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.307459 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.307466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:48.307651 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.307676 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 22:41:48.307792 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 22:41:48.307945 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":89,"thread_start_us":351,"threads_started":1}
I20260806 22:41:48.309285 2240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7808e1cf4384423981daacea7ae7af3a" candidate_uuid: "0a62863097244864bed64a9809dfdcde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9517100acaf74cc3a6b7caf88c305b03" is_pre_election: true
I20260806 22:41:48.309921 2240 raft_consensus.cc:2454] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a62863097244864bed64a9809dfdcde in term 0.
I20260806 22:41:48.309983 1800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7808e1cf4384423981daacea7ae7af3a" candidate_uuid: "0a62863097244864bed64a9809dfdcde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37baa871ee4ebbb82552241b0069b2" is_pre_election: true
I20260806 22:41:48.310514 1800 raft_consensus.cc:2454] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a62863097244864bed64a9809dfdcde in term 0.
I20260806 22:41:48.310675 1930 leader_election.cc:302] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a62863097244864bed64a9809dfdcde, 9517100acaf74cc3a6b7caf88c305b03; no voters:
I20260806 22:41:48.311151 2376 raft_consensus.cc:2781] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:48.311285 2376 raft_consensus.cc:482] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:48.311435 2376 raft_consensus.cc:3037] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:48.313773 2376 raft_consensus.cc:504] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:48.314453 2376 leader_election.cc:288] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [CANDIDATE]: Term 1 election: Requested vote from peers cd37baa871ee4ebbb82552241b0069b2 (127.23.100.129:34075), 9517100acaf74cc3a6b7caf88c305b03 (127.23.100.131:34259)
I20260806 22:41:48.315019 2240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7808e1cf4384423981daacea7ae7af3a" candidate_uuid: "0a62863097244864bed64a9809dfdcde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9517100acaf74cc3a6b7caf88c305b03"
I20260806 22:41:48.315032 1800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7808e1cf4384423981daacea7ae7af3a" candidate_uuid: "0a62863097244864bed64a9809dfdcde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd37baa871ee4ebbb82552241b0069b2"
I20260806 22:41:48.315251 2240 raft_consensus.cc:3037] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:48.315275 1800 raft_consensus.cc:3037] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:48.317659 2240 raft_consensus.cc:2454] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a62863097244864bed64a9809dfdcde in term 1.
I20260806 22:41:48.318109 1930 leader_election.cc:302] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a62863097244864bed64a9809dfdcde, 9517100acaf74cc3a6b7caf88c305b03; no voters:
I20260806 22:41:48.318140 1800 raft_consensus.cc:2454] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a62863097244864bed64a9809dfdcde in term 1.
I20260806 22:41:48.318498 2376 raft_consensus.cc:2781] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:48.319015 2376 raft_consensus.cc:686] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 1 LEADER]: Becoming Leader. State: Replica: 0a62863097244864bed64a9809dfdcde, State: Running, Role: LEADER
I20260806 22:41:48.319577 2376 consensus_queue.cc:237] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:48.324965 357 catalog_manager.cc:5495] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a62863097244864bed64a9809dfdcde (127.23.100.130). New cstate: current_term: 1 leader_uuid: "0a62863097244864bed64a9809dfdcde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } health_report { overall_health: HEALTHY } } }
I20260806 22:41:48.363960 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.358876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.23.100.130:46003)
0806 22:41:48.359135 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.359140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.359158 (+ 18us) server_negotiation.cc:407] Connection header received
0806 22:41:48.359331 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.359335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.359414 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.359545 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:48.359891 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.360559 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.361980 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.362393 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.362502 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.362897 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:48.363378 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:48.363507 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.363578 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.363646 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 22:41:48.372776 2240 raft_consensus.cc:1261] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03 [term 1 FOLLOWER]: Refusing update from remote peer 0a62863097244864bed64a9809dfdcde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:48.372748 1800 raft_consensus.cc:1261] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2 [term 1 FOLLOWER]: Refusing update from remote peer 0a62863097244864bed64a9809dfdcde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:48.373859 2384 consensus_queue.cc:1035] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [LEADER]: Connected to new peer: Peer: permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:48.374311 2376 consensus_queue.cc:1035] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:48.388136 2387 mvcc.cc:205] Tried to move back new op lower bound from 7315685819884158976 to 7315685819680980992. Current Snapshot: MvccSnapshot[applied={T|T < 7315685819884158976}]
I20260806 22:41:48.389173 2390 mvcc.cc:205] Tried to move back new op lower bound from 7315685819884158976 to 7315685819680980992. Current Snapshot: MvccSnapshot[applied={T|T < 7315685819884158976}]
I20260806 22:41:48.404835 357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:48.405090 357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:48.412029 357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a9e131374e743a4803b662ca72cf6dd: Sending DeleteTablet for 3 replicas of tablet 7808e1cf4384423981daacea7ae7af3a
I20260806 22:41:48.413334 2208 tablet_service.cc:1514] Processing DeleteTablet for tablet 7808e1cf4384423981daacea7ae7af3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:48 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:38660
I20260806 22:41:48.415045 1767 tablet_service.cc:1514] Processing DeleteTablet for tablet 7808e1cf4384423981daacea7ae7af3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:48 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:44958
I20260806 22:41:48.416071 1997 tablet_service.cc:1514] Processing DeleteTablet for tablet 7808e1cf4384423981daacea7ae7af3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:48 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:39636
I20260806 22:41:48.427680 2404 tablet_replica.cc:330] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: stopping tablet replica
I20260806 22:41:48.428911 2404 raft_consensus.cc:2229] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:48.430246 2404 raft_consensus.cc:2258] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:48.431991 2403 tablet_replica.cc:330] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: stopping tablet replica
I20260806 22:41:48.432461 2403 raft_consensus.cc:2229] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:48.432955 2403 raft_consensus.cc:2258] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:48.435214 2404 ts_tablet_manager.cc:1905] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:48.435367 2403 ts_tablet_manager.cc:1905] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:48.444196 2403 ts_tablet_manager.cc:1918] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:48.444478 2403 log.cc:1196] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/7808e1cf4384423981daacea7ae7af3a
I20260806 22:41:48.445308 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 310
I20260806 22:41:48.446697 396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:48.446743 2403 ts_tablet_manager.cc:1939] T 7808e1cf4384423981daacea7ae7af3a P 9517100acaf74cc3a6b7caf88c305b03: Deleting consensus metadata
I20260806 22:41:48.447723 2404 ts_tablet_manager.cc:1918] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:48.447922 2404 log.cc:1196] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/7808e1cf4384423981daacea7ae7af3a
I20260806 22:41:48.450152 2404 ts_tablet_manager.cc:1939] T 7808e1cf4384423981daacea7ae7af3a P cd37baa871ee4ebbb82552241b0069b2: Deleting consensus metadata
I20260806 22:41:48.454783 2407 tablet_replica.cc:330] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde: stopping tablet replica
I20260806 22:41:48.455428 2407 raft_consensus.cc:2229] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:48.456231 2407 raft_consensus.cc:2258] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:48.462855 2407 ts_tablet_manager.cc:1905] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:48.469914 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.417728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.23.100.189:37055)
0806 22:41:48.417917 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.417923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.444204 (+ 26281us) server_negotiation.cc:407] Connection header received
0806 22:41:48.444832 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.444839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.444938 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.445060 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:48.448444 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.449786 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.453069 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.453931 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.454048 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.462924 (+ 8876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:48.462932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:48.462938 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:48.462986 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:48.466602 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.467597 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.467604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.467609 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.467760 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.468208 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.468214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.468218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.468596 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:48.468847 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.469251 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.469500 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 22:41:48.470117 2407 ts_tablet_manager.cc:1918] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:48.470281 2407 log.cc:1196] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/7808e1cf4384423981daacea7ae7af3a
I20260806 22:41:48.471983 2407 ts_tablet_manager.cc:1939] T 7808e1cf4384423981daacea7ae7af3a P 0a62863097244864bed64a9809dfdcde: Deleting consensus metadata
I20260806 22:41:48.487528 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.444547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.23.100.188:46107)
0806 22:41:48.444744 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.444750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.470487 (+ 25737us) server_negotiation.cc:407] Connection header received
0806 22:41:48.470757 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.470763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.470857 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.470937 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:48.473068 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.474444 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.476358 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.477024 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.477147 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.480652 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:48.480658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:48.480663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:48.480706 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:48.484358 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.485255 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.485261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.485265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.485413 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.485849 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.485855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.485865 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.486213 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:48.486450 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.486844 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.487080 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 22:41:48.595100 336 catalog_manager.cc:4841] TS 9517100acaf74cc3a6b7caf88c305b03 (127.23.100.131:34259): tablet 7808e1cf4384423981daacea7ae7af3a (table test-table [id=a4ee016c79c6421f9337d44cff4ff312]) successfully deleted
W20260806 22:41:48.595520 336 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.23.100.131:34259, user_credentials={real_user=slave}} blocked reactor thread for 146069us
I20260806 22:41:48.595901 336 catalog_manager.cc:4841] TS cd37baa871ee4ebbb82552241b0069b2 (127.23.100.129:34075): tablet 7808e1cf4384423981daacea7ae7af3a (table test-table [id=a4ee016c79c6421f9337d44cff4ff312]) successfully deleted
I20260806 22:41:48.595113 334 catalog_manager.cc:4841] TS 0a62863097244864bed64a9809dfdcde (127.23.100.130:46003): tablet 7808e1cf4384423981daacea7ae7af3a (table test-table [id=a4ee016c79c6421f9337d44cff4ff312]) successfully deleted
I20260806 22:41:48.596673 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 310
I20260806 22:41:48.649156 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.631647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.23.100.189:37055)
0806 22:41:48.631907 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.631917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.632922 (+ 1005us) server_negotiation.cc:407] Connection header received
0806 22:41:48.633127 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.633136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.633243 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.633355 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:48.635219 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.636364 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.638398 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.638996 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.639143 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.642594 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:48.642600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:48.642603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:48.642637 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:48.645714 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.646708 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.646715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.646719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.646897 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.647405 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.647413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.647418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.647832 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:48.648105 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.648525 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.648766 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 22:41:48.652031 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:48.633819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.23.100.188:46107)
0806 22:41:48.634026 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:48.634032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:48.635463 (+ 1431us) server_negotiation.cc:407] Connection header received
0806 22:41:48.635596 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:48.635602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:48.635696 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:48.635806 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:48.638091 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.639208 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.641255 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:48.642070 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:48.642222 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:48.645663 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:48.645673 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:48.645678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:48.645723 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:48.649172 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.649978 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.649984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.649988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.650116 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:48.650575 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:48.650579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:48.650581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:48.650884 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:48.651136 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:48.651487 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 22:41:48.651707 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 22:41:49.396621 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.395592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.131:51149)
0806 22:41:49.396201 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:49.396350 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":333,"threads_started":1}
W20260806 22:41:49.397122 2319 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:45503 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
I20260806 22:41:49.399585 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.398593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.129:59803)
0806 22:41:49.399168 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:49.399347 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":286,"threads_started":1}
W20260806 22:41:49.400329 1871 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:45503 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
I20260806 22:41:49.400637 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.398965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.130:37699)
0806 22:41:49.400000 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:49.400161 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":1214,"threads_started":1}
W20260806 22:41:49.401221 2090 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:45503 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
I20260806 22:41:49.912712 2426 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a9e131374e743a4803b662ca72cf6dd)
I20260806 22:41:49.913091 2426 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } }
I20260806 22:41:49.914553 2427 raft_consensus.cc:482] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a9e131374e743a4803b662ca72cf6dd)
I20260806 22:41:49.914829 2426 leader_election.cc:288] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a9e131374e743a4803b662ca72cf6dd (127.23.100.190:45503), e04600f32f8c41ed812bcc5bd9332e28 (127.23.100.189:37055)
I20260806 22:41:49.916036 2427 raft_consensus.cc:504] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } }
I20260806 22:41:49.916536 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.914956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:53178)
0806 22:41:49.916000 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:49.916264 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":1083,"threads_started":1}
W20260806 22:41:49.916786 1569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
I20260806 22:41:49.918874 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.918423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:53198)
0806 22:41:49.918601 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:49.918687 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 22:41:49.919129 1569 leader_election.cc:334] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a9e131374e743a4803b662ca72cf6dd (127.23.100.190:45503): Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
I20260806 22:41:49.919894 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.917932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:53190)
0806 22:41:49.918866 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:49.919531 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":694,"threads_started":1}
W20260806 22:41:49.920264 1279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
I20260806 22:41:49.920509 2427 leader_election.cc:288] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a9e131374e743a4803b662ca72cf6dd (127.23.100.190:45503), a2153cc3e61143849fa047c85042a332 (127.23.100.188:46107)
I20260806 22:41:49.922300 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.921802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:53200)
0806 22:41:49.922003 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:49.922099 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 22:41:49.922552 1279 leader_election.cc:334] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a9e131374e743a4803b662ca72cf6dd (127.23.100.190:45503): Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
I20260806 22:41:49.923820 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.915066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:37055 (local address 127.0.0.1:37278)
0806 22:41:49.916351 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:49.916375 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:49.916546 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:49.917143 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:49.917147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:49.917168 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:49.917405 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:49.917412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:49.919003 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:49.919010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:49.921277 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:49.921293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:49.922965 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:49.922974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:49.923149 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:49.923175 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 22:41:49.923250 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 22:41:49.923348 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"thread_start_us":635,"threads_started":1}
I20260806 22:41:49.923851 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.915136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.23.100.189:37055)
0806 22:41:49.915860 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:49.915870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:49.916463 (+ 593us) server_negotiation.cc:407] Connection header received
0806 22:41:49.916737 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:49.916743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:49.916829 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:49.916890 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:49.917590 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:49.918808 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:49.921473 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:49.922776 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:49.922943 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:49.923029 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:49.923311 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 22:41:49.923465 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":278,"threads_started":1}
I20260806 22:41:49.924561 1311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2153cc3e61143849fa047c85042a332" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e04600f32f8c41ed812bcc5bd9332e28" is_pre_election: true
I20260806 22:41:49.924934 1311 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2153cc3e61143849fa047c85042a332 in term 1.
I20260806 22:41:49.925555 1561 leader_election.cc:302] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2153cc3e61143849fa047c85042a332, e04600f32f8c41ed812bcc5bd9332e28; no voters: 9a9e131374e743a4803b662ca72cf6dd
I20260806 22:41:49.926038 2426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 22:41:49.926205 2426 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a9e131374e743a4803b662ca72cf6dd)
I20260806 22:41:49.926317 2426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 1 FOLLOWER]: Advancing to term 2
I20260806 22:41:49.928035 2436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.919343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46107 (local address 127.0.0.1:36376)
0806 22:41:49.919919 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:49.919943 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:49.920067 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:49.920960 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:49.920966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:49.920996 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:49.921394 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:49.921413 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:49.923074 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:49.923084 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:49.925399 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:49.925415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:49.927339 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:49.927345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:49.927563 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:49.927591 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 22:41:49.927676 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 22:41:49.927779 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":653,"threads_started":1}
I20260806 22:41:49.928370 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.919832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.23.100.188:46107)
0806 22:41:49.920619 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:49.920624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:49.920646 (+ 22us) server_negotiation.cc:407] Connection header received
0806 22:41:49.920712 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:49.920717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:49.920798 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:49.921474 (+ 676us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:49.921722 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:49.922924 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:49.925653 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:49.927079 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:49.927332 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:49.927411 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:49.927737 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 22:41:49.927890 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":527,"threads_started":1}
I20260806 22:41:49.928936 1606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e04600f32f8c41ed812bcc5bd9332e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a2153cc3e61143849fa047c85042a332" is_pre_election: true
I20260806 22:41:49.929603 2426 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } }
I20260806 22:41:49.930086 1606 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e04600f32f8c41ed812bcc5bd9332e28 in current term 2: Already voted for candidate a2153cc3e61143849fa047c85042a332 in this term.
I20260806 22:41:49.930341 2426 leader_election.cc:288] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [CANDIDATE]: Term 2 election: Requested vote from peers 9a9e131374e743a4803b662ca72cf6dd (127.23.100.190:45503), e04600f32f8c41ed812bcc5bd9332e28 (127.23.100.189:37055)
I20260806 22:41:49.930929 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.930440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:53214)
0806 22:41:49.930680 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:49.930790 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 22:41:49.930886 1311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2153cc3e61143849fa047c85042a332" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e04600f32f8c41ed812bcc5bd9332e28"
I20260806 22:41:49.930894 1275 leader_election.cc:302] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e04600f32f8c41ed812bcc5bd9332e28; no voters: 9a9e131374e743a4803b662ca72cf6dd, a2153cc3e61143849fa047c85042a332
I20260806 22:41:49.931203 1311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 1 FOLLOWER]: Advancing to term 2
I20260806 22:41:49.932096 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:49.931589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:53216)
0806 22:41:49.931773 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:49.931913 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260806 22:41:49.932358 1569 leader_election.cc:334] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a9e131374e743a4803b662ca72cf6dd (127.23.100.190:45503): Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
I20260806 22:41:49.934413 1311 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2153cc3e61143849fa047c85042a332 in term 2.
I20260806 22:41:49.934656 2427 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 22:41:49.934886 1561 leader_election.cc:302] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2153cc3e61143849fa047c85042a332, e04600f32f8c41ed812bcc5bd9332e28; no voters: 9a9e131374e743a4803b662ca72cf6dd
I20260806 22:41:49.935217 2426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 22:41:49.935657 2426 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [term 2 LEADER]: Becoming Leader. State: Replica: a2153cc3e61143849fa047c85042a332, State: Running, Role: LEADER
I20260806 22:41:49.936067 2426 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } }
I20260806 22:41:49.938273 2440 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2153cc3e61143849fa047c85042a332. Latest consensus state: current_term: 2 leader_uuid: "a2153cc3e61143849fa047c85042a332" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:49.938731 2440 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:49.939658 2442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:49.943486 2442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:49.944365 2442 catalog_manager.cc:1259] Loaded cluster ID: 9743b2c273644d7899b4f95bc4699be9
I20260806 22:41:49.944440 2442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:49.945734 2442 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:49.946630 2442 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Loaded TSK: 0
I20260806 22:41:49.947579 2442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:41:49.995677 1584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:50.003290 1584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:50.008128 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.007678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:53252)
0806 22:41:50.007881 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.007967 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 22:41:50.008204 1311 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [term 2 FOLLOWER]: Refusing update from remote peer a2153cc3e61143849fa047c85042a332: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 22:41:50.009132 2426 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 22:41:50.009438 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.008995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:53268)
0806 22:41:50.009194 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.009273 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 22:41:50.009642 1569 consensus_peers.cc:487] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 -> Peer 9a9e131374e743a4803b662ca72cf6dd (127.23.100.190:45503): Couldn't send request to peer 9a9e131374e743a4803b662ca72cf6dd. Status: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 22:41:50.012346 2427 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2153cc3e61143849fa047c85042a332. Latest consensus state: current_term: 2 leader_uuid: "a2153cc3e61143849fa047c85042a332" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:50.012637 2427 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:41:50.014437 2440 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a2153cc3e61143849fa047c85042a332" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:50.014746 2440 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:50.015304 2447 mvcc.cc:205] Tried to move back new op lower bound from 7315685826585591808 to 7315685826302029824. Current Snapshot: MvccSnapshot[applied={T|T < 7315685826585591808}]
I20260806 22:41:50.015198 2427 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a2153cc3e61143849fa047c85042a332" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9a9e131374e743a4803b662ca72cf6dd" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45503 } } peers { permanent_uuid: "e04600f32f8c41ed812bcc5bd9332e28" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 37055 } attrs { promote: false } } peers { permanent_uuid: "a2153cc3e61143849fa047c85042a332" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46107 } attrs { promote: false } } }
I20260806 22:41:50.016258 2427 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04600f32f8c41ed812bcc5bd9332e28 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:41:50.044117 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.024523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:34259 (local address 127.0.0.1:38676)
0806 22:41:50.024739 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.024765 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:50.024910 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:50.038329 (+ 13419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:50.038336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:50.038382 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:50.038639 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:50.038656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.040041 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.040047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:50.041496 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:50.041541 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.043043 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.043050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:50.043337 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.043386 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 22:41:50.043480 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 22:41:50.043671 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 22:41:50.045320 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.024812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.23.100.131:34259)
0806 22:41:50.037800 (+ 12988us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:50.037810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:50.037842 (+ 32us) server_negotiation.cc:407] Connection header received
0806 22:41:50.037948 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:50.037957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:50.038094 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:50.038837 (+ 743us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:50.039069 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.039865 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.041886 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.042835 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.044666 (+ 1831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.044760 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:50.044827 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 22:41:50.044998 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12656,"thread_start_us":12722,"threads_started":1}
I20260806 22:41:50.046370 2208 tablet_service.cc:1467] Processing CreateTablet for tablet 2e96463a70fe436f9f930248e23dab7c (DEFAULT_TABLE table=test-table [id=3b72e9dff31c4d609c5a0f5e882db3a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:50.047061 2208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e96463a70fe436f9f930248e23dab7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:50.054461 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.025283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:34075 (local address 127.0.0.1:44972)
0806 22:41:50.025497 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.025541 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:50.025665 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:50.032449 (+ 6784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:50.032459 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:50.032499 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:50.032977 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:50.032995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.037140 (+ 4145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.037151 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:50.040196 (+ 3045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:50.040220 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.045628 (+ 5408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.045638 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:50.045888 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.045959 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 22:41:50.046069 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 22:41:50.053969 (+ 7900us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 22:41:50.063036 2460 tablet_bootstrap.cc:492] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03: Bootstrap starting.
I20260806 22:41:50.065261 2460 tablet_bootstrap.cc:654] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:50.066660 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.026038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44972 (local address 127.23.100.129:34075)
0806 22:41:50.031158 (+ 5120us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:50.031167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:50.031195 (+ 28us) server_negotiation.cc:407] Connection header received
0806 22:41:50.031297 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:50.031306 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:50.031416 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:50.033196 (+ 1780us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:50.033411 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.035388 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.040513 (+ 5125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.042224 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.065806 (+ 23582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.065936 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:50.066023 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 22:41:50.066204 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4939,"thread_start_us":4788,"threads_started":1}
I20260806 22:41:50.067837 1767 tablet_service.cc:1467] Processing CreateTablet for tablet 2e96463a70fe436f9f930248e23dab7c (DEFAULT_TABLE table=test-table [id=3b72e9dff31c4d609c5a0f5e882db3a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:50.068560 1767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e96463a70fe436f9f930248e23dab7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:50.081547 2460 tablet_bootstrap.cc:492] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03: No bootstrap required, opened a new log
I20260806 22:41:50.081756 2460 ts_tablet_manager.cc:1397] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20260806 22:41:50.082660 2462 tablet_bootstrap.cc:492] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2: Bootstrap starting.
I20260806 22:41:50.082782 2460 raft_consensus.cc:348] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:50.083069 2460 raft_consensus.cc:374] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:50.083184 2460 raft_consensus.cc:729] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9517100acaf74cc3a6b7caf88c305b03, State: Initialized, Role: FOLLOWER
I20260806 22:41:50.083632 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.025469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:46003 (local address 127.0.0.1:39654)
0806 22:41:50.038573 (+ 13104us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.038603 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:50.038732 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:50.071229 (+ 32497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:50.071235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:50.071271 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:50.071673 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:50.071688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.078373 (+ 6685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.078380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:50.080438 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:50.080455 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.082670 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.082678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:50.082919 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.082962 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 22:41:50.083051 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 22:41:50.083241 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12930,"thread_start_us":13321,"threads_started":1}
I20260806 22:41:50.083469 2460 consensus_queue.cc:260] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:50.084579 2460 ts_tablet_manager.cc:1428] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:50.084882 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.026022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.23.100.130:46003)
0806 22:41:50.070818 (+ 44796us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:50.070827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:50.070856 (+ 29us) server_negotiation.cc:407] Connection header received
0806 22:41:50.070943 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:50.070949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:50.071039 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:50.071177 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:50.076858 (+ 5681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.078179 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.081021 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.082389 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.084160 (+ 1771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.084254 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:50.084315 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 22:41:50.084502 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44625,"thread_start_us":44485,"threads_started":1}
I20260806 22:41:50.086041 2462 tablet_bootstrap.cc:654] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:50.086401 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.085287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.131:49037)
0806 22:41:50.085985 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.086139 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":340,"threads_started":1}
I20260806 22:41:50.086323 1997 tablet_service.cc:1467] Processing CreateTablet for tablet 2e96463a70fe436f9f930248e23dab7c (DEFAULT_TABLE table=test-table [id=3b72e9dff31c4d609c5a0f5e882db3a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:50.086839 1997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e96463a70fe436f9f930248e23dab7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:50.088924 2462 tablet_bootstrap.cc:492] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2: No bootstrap required, opened a new log
I20260806 22:41:50.089077 2462 ts_tablet_manager.cc:1397] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 22:41:50.090076 2462 raft_consensus.cc:348] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:50.090374 2462 raft_consensus.cc:374] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:50.090515 2462 raft_consensus.cc:729] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd37baa871ee4ebbb82552241b0069b2, State: Initialized, Role: FOLLOWER
I20260806 22:41:50.090816 2462 consensus_queue.cc:260] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:50.092042 2462 ts_tablet_manager.cc:1428] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:50.094085 1869 heartbeater.cc:503] Master 127.23.100.188:46107 was elected leader, sending a full tablet report...
I20260806 22:41:50.099831 2318 heartbeater.cc:503] Master 127.23.100.188:46107 was elected leader, sending a full tablet report...
I20260806 22:41:50.100046 2474 tablet_bootstrap.cc:492] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde: Bootstrap starting.
I20260806 22:41:50.102913 2474 tablet_bootstrap.cc:654] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:50.104682 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.104024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.131:37703)
0806 22:41:50.104392 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.104506 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260806 22:41:50.107407 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.092136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.129:44269)
0806 22:41:50.106776 (+ 14640us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.107014 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14454,"thread_start_us":14336,"threads_started":1}
I20260806 22:41:50.108999 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.108427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.129:43175)
0806 22:41:50.108672 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.108797 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 22:41:50.111552 2474 tablet_bootstrap.cc:492] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde: No bootstrap required, opened a new log
I20260806 22:41:50.111742 2474 ts_tablet_manager.cc:1397] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260806 22:41:50.112526 2474 raft_consensus.cc:348] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:50.112756 2474 raft_consensus.cc:374] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:50.112860 2474 raft_consensus.cc:729] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a62863097244864bed64a9809dfdcde, State: Initialized, Role: FOLLOWER
I20260806 22:41:50.113149 2474 consensus_queue.cc:260] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:50.114049 2474 ts_tablet_manager.cc:1428] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260806 22:41:50.115468 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.114394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.130:34051)
0806 22:41:50.114978 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.115160 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":306,"threads_started":1}
I20260806 22:41:50.116441 2088 heartbeater.cc:503] Master 127.23.100.188:46107 was elected leader, sending a full tablet report...
I20260806 22:41:50.116971 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.116521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.130:44103)
0806 22:41:50.116726 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.116841 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 22:41:50.150653 2472 raft_consensus.cc:482] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:50.150916 2472 raft_consensus.cc:504] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:50.152465 2472 leader_election.cc:288] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9517100acaf74cc3a6b7caf88c305b03 (127.23.100.131:34259), 0a62863097244864bed64a9809dfdcde (127.23.100.130:46003)
I20260806 22:41:50.160476 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.153042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:51509 (local address 127.23.100.130:46003)
0806 22:41:50.153282 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:50.153292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:50.153861 (+ 569us) server_negotiation.cc:407] Connection header received
0806 22:41:50.154021 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:50.154025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:50.154086 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:50.154157 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:50.154672 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.155633 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.158125 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.159401 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.159559 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.159634 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:50.160063 (+ 429us) server_negotiation.cc:299] Negotiation successful
0806 22:41:50.160216 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 22:41:50.160672 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.152830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:54289 (local address 127.23.100.131:34259)
0806 22:41:50.153043 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:50.153049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:50.153181 (+ 132us) server_negotiation.cc:407] Connection header received
0806 22:41:50.153464 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:50.153470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:50.153593 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:50.153696 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:50.154274 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.155340 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.158147 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.159394 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.159565 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.159634 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:50.160284 (+ 650us) server_negotiation.cc:299] Negotiation successful
0806 22:41:50.160424 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 22:41:50.160756 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.152737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:34259 (local address 127.23.100.129:54289)
0806 22:41:50.153050 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.153079 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:50.153259 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:50.153743 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:50.153750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:50.153783 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:50.154094 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:50.154105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.155497 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.155501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:50.157950 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:50.157967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.159829 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.159835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:50.160097 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.160131 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 22:41:50.160225 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 22:41:50.160367 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":24}
I20260806 22:41:50.161093 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.152942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:46003 (local address 127.23.100.129:51509)
0806 22:41:50.153733 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.153750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:50.153874 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:50.154204 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:50.154208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:50.154235 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:50.154509 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:50.154518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.155760 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.155768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:50.157950 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:50.157967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.159627 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.159633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:50.159889 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.159917 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 22:41:50.159999 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 22:41:50.160106 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":27,"thread_start_us":416,"threads_started":1}
I20260806 22:41:50.161612 2240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e96463a70fe436f9f930248e23dab7c" candidate_uuid: "cd37baa871ee4ebbb82552241b0069b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9517100acaf74cc3a6b7caf88c305b03" is_pre_election: true
I20260806 22:41:50.161871 2240 raft_consensus.cc:2454] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37baa871ee4ebbb82552241b0069b2 in term 0.
I20260806 22:41:50.162142 2025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e96463a70fe436f9f930248e23dab7c" candidate_uuid: "cd37baa871ee4ebbb82552241b0069b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a62863097244864bed64a9809dfdcde" is_pre_election: true
I20260806 22:41:50.162446 1699 leader_election.cc:302] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9517100acaf74cc3a6b7caf88c305b03, cd37baa871ee4ebbb82552241b0069b2; no voters:
I20260806 22:41:50.162567 2025 raft_consensus.cc:2454] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd37baa871ee4ebbb82552241b0069b2 in term 0.
I20260806 22:41:50.162849 2472 raft_consensus.cc:2781] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:50.162989 2472 raft_consensus.cc:482] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:50.163093 2472 raft_consensus.cc:3037] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:50.165293 2472 raft_consensus.cc:504] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:50.166059 2472 leader_election.cc:288] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [CANDIDATE]: Term 1 election: Requested vote from peers 9517100acaf74cc3a6b7caf88c305b03 (127.23.100.131:34259), 0a62863097244864bed64a9809dfdcde (127.23.100.130:46003)
I20260806 22:41:50.166581 2240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e96463a70fe436f9f930248e23dab7c" candidate_uuid: "cd37baa871ee4ebbb82552241b0069b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9517100acaf74cc3a6b7caf88c305b03"
I20260806 22:41:50.166651 2025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e96463a70fe436f9f930248e23dab7c" candidate_uuid: "cd37baa871ee4ebbb82552241b0069b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a62863097244864bed64a9809dfdcde"
I20260806 22:41:50.166798 2240 raft_consensus.cc:3037] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:50.166971 2025 raft_consensus.cc:3037] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:50.168883 2240 raft_consensus.cc:2454] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37baa871ee4ebbb82552241b0069b2 in term 1.
I20260806 22:41:50.169396 1699 leader_election.cc:302] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9517100acaf74cc3a6b7caf88c305b03, cd37baa871ee4ebbb82552241b0069b2; no voters:
I20260806 22:41:50.169534 2025 raft_consensus.cc:2454] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd37baa871ee4ebbb82552241b0069b2 in term 1.
I20260806 22:41:50.169786 2472 raft_consensus.cc:2781] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:50.170401 2472 raft_consensus.cc:686] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 1 LEADER]: Becoming Leader. State: Replica: cd37baa871ee4ebbb82552241b0069b2, State: Running, Role: LEADER
I20260806 22:41:50.170840 2472 consensus_queue.cc:237] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } } peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } }
I20260806 22:41:50.171803 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.171318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.129:54867)
0806 22:41:50.171501 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.171629 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 22:41:50.173979 2097 tablet.cc:2369] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:41:50.174793 1584 catalog_manager.cc:5495] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd37baa871ee4ebbb82552241b0069b2 (127.23.100.129). New cstate: current_term: 1 leader_uuid: "cd37baa871ee4ebbb82552241b0069b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd37baa871ee4ebbb82552241b0069b2" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 34075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 } health_report { overall_health: UNKNOWN } } }
I20260806 22:41:50.216418 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.211210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.23.100.129:34075)
0806 22:41:50.211473 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:50.211481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:50.211502 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:41:50.211602 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:50.211606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:50.211683 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:50.211843 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:50.212216 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.212917 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.214262 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.214697 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.214835 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.215264 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:50.215795 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:50.215963 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:50.216062 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 22:41:50.216145 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 22:41:50.220948 2240 raft_consensus.cc:1261] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03 [term 1 FOLLOWER]: Refusing update from remote peer cd37baa871ee4ebbb82552241b0069b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:50.221266 2025 raft_consensus.cc:1261] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde [term 1 FOLLOWER]: Refusing update from remote peer cd37baa871ee4ebbb82552241b0069b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:50.221845 2479 consensus_queue.cc:1035] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9517100acaf74cc3a6b7caf88c305b03" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:50.222368 2472 consensus_queue.cc:1035] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a62863097244864bed64a9809dfdcde" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:50.225167 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.224500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.131:34713)
0806 22:41:50.224746 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.224901 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260806 22:41:50.226418 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.225063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.130:45417)
0806 22:41:50.225995 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.226145 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":691}
I20260806 22:41:50.230645 2482 mvcc.cc:205] Tried to move back new op lower bound from 7315685827457318912 to 7315685827264020480. Current Snapshot: MvccSnapshot[applied={T|T < 7315685827457318912}]
I20260806 22:41:50.231963 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.231555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.131:49391)
0806 22:41:50.231773 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.231847 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 22:41:50.235705 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.235142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.130:53383)
0806 22:41:50.235417 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.235543 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260806 22:41:50.236878 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.235908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.129:43817)
0806 22:41:50.236527 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.236641 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506}
I20260806 22:41:50.240723 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.240131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.23.100.129:34601)
0806 22:41:50.240458 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.240568 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236}
I20260806 22:41:50.249914 1584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:50.250170 1584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:50.255800 1584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2153cc3e61143849fa047c85042a332: Sending DeleteTablet for 3 replicas of tablet 2e96463a70fe436f9f930248e23dab7c
I20260806 22:41:50.257196 2208 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e96463a70fe436f9f930248e23dab7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:50 UTC) from {username='oryx', principal='oryx/127.23.100.188@KRBTEST.COM'} at 127.0.0.1:38676
I20260806 22:41:50.257395 1997 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e96463a70fe436f9f930248e23dab7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:50 UTC) from {username='oryx', principal='oryx/127.23.100.188@KRBTEST.COM'} at 127.0.0.1:39654
I20260806 22:41:50.258227 1767 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e96463a70fe436f9f930248e23dab7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:50 UTC) from {username='oryx', principal='oryx/127.23.100.188@KRBTEST.COM'} at 127.0.0.1:44972
I20260806 22:41:50.258704 2500 tablet_replica.cc:330] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde: stopping tablet replica
I20260806 22:41:50.258970 2503 tablet_replica.cc:330] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2: stopping tablet replica
I20260806 22:41:50.258924 2500 raft_consensus.cc:2229] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:50.259143 2503 raft_consensus.cc:2229] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:50.259265 2500 raft_consensus.cc:2258] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:50.259748 2503 raft_consensus.cc:2258] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:50.259888 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 1551
I20260806 22:41:50.260543 2500 ts_tablet_manager.cc:1905] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:50.260928 2503 ts_tablet_manager.cc:1905] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:50.267491 2500 ts_tablet_manager.cc:1918] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:50.267547 2503 ts_tablet_manager.cc:1918] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:50.267673 2500 log.cc:1196] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/2e96463a70fe436f9f930248e23dab7c
I20260806 22:41:50.267787 2503 log.cc:1196] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/2e96463a70fe436f9f930248e23dab7c
I20260806 22:41:50.268225 2500 ts_tablet_manager.cc:1939] T 2e96463a70fe436f9f930248e23dab7c P 0a62863097244864bed64a9809dfdcde: Deleting consensus metadata
I20260806 22:41:50.268334 2503 ts_tablet_manager.cc:1939] T 2e96463a70fe436f9f930248e23dab7c P cd37baa871ee4ebbb82552241b0069b2: Deleting consensus metadata
I20260806 22:41:50.269564 1569 catalog_manager.cc:4841] TS cd37baa871ee4ebbb82552241b0069b2 (127.23.100.129:34075): tablet 2e96463a70fe436f9f930248e23dab7c (table test-table [id=3b72e9dff31c4d609c5a0f5e882db3a3]) successfully deleted
I20260806 22:41:50.269541 1563 catalog_manager.cc:4841] TS 0a62863097244864bed64a9809dfdcde (127.23.100.130:46003): tablet 2e96463a70fe436f9f930248e23dab7c (table test-table [id=3b72e9dff31c4d609c5a0f5e882db3a3]) successfully deleted
I20260806 22:41:50.270157 2498 tablet_replica.cc:330] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03: stopping tablet replica
I20260806 22:41:50.270368 2498 raft_consensus.cc:2229] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:50.270659 2498 raft_consensus.cc:2258] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:50.271957 2498 ts_tablet_manager.cc:1905] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:50.277230 2498 ts_tablet_manager.cc:1918] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:50.277369 2498 log.cc:1196] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/2e96463a70fe436f9f930248e23dab7c
I20260806 22:41:50.277869 2498 ts_tablet_manager.cc:1939] T 2e96463a70fe436f9f930248e23dab7c P 9517100acaf74cc3a6b7caf88c305b03: Deleting consensus metadata
I20260806 22:41:50.279085 1569 catalog_manager.cc:4841] TS 9517100acaf74cc3a6b7caf88c305b03 (127.23.100.131:34259): tablet 2e96463a70fe436f9f930248e23dab7c (table test-table [id=3b72e9dff31c4d609c5a0f5e882db3a3]) successfully deleted
I20260806 22:41:50.287107 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.263564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.23.100.129:34075)
0806 22:41:50.263836 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:50.263843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:50.271545 (+ 7702us) server_negotiation.cc:407] Connection header received
0806 22:41:50.271613 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:50.271618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:50.271737 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:50.271841 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:50.273790 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.274919 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.276841 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.277578 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.277722 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.280841 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:50.280863 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:50.280879 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:50.280910 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:50.284081 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:50.284806 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:50.284814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:50.284816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:50.284979 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:50.285357 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:50.285367 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:50.285372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:50.285771 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:50.286099 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:50.286404 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 22:41:50.286629 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260806 22:41:50.287846 1863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:50.392879 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 1551
I20260806 22:41:50.418114 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 1891
I20260806 22:41:50.434093 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.418945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46246 (local address 127.23.100.130:46003)
0806 22:41:50.419169 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:50.419175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:50.419289 (+ 114us) server_negotiation.cc:407] Connection header received
0806 22:41:50.419540 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:50.419545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:50.419669 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:50.419772 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:50.421447 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.422507 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.423821 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.424411 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.424560 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.428101 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:50.428118 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:50.428134 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:50.428165 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:50.431348 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:50.432052 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:50.432060 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:50.432062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:50.432196 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:50.432526 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:50.432529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:50.432531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:50.432894 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:50.433106 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:50.433469 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 22:41:50.433722 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 22:41:50.434832 2082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:50.551517 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.551084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:39102)
0806 22:41:50.551259 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.551352 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 22:41:50.552891 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.552509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:45503 (local address 127.0.0.1:39104)
0806 22:41:50.552671 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:50.552767 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 22:41:50.561157 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 1891
I20260806 22:41:50.593627 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 2111
I20260806 22:41:50.607658 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:50.594566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42160 (local address 127.23.100.131:34259)
0806 22:41:50.594779 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:50.594784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:50.594805 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:41:50.595007 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:50.595012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:50.595089 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:50.595186 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:50.596615 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.597455 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.598878 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:50.599315 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:50.599437 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:50.601980 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:50.601995 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:50.602012 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:50.602058 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:50.604867 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:50.605635 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:50.605643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:50.605646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:50.605778 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:50.606118 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:50.606121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:50.606123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:50.606494 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:50.606721 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:50.607038 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 22:41:50.607359 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 22:41:50.608392 2306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:50.704388 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 2111
I20260806 22:41:50.730814 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 1086
I20260806 22:41:50.732041 1339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:50.852670 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 1086
I20260806 22:41:50.883074 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 1248
I20260806 22:41:50.884256 1636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:51.007254 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 1248
2026-08-06T22:41:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10192 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:41:51 dist-test-slave-zr1t krb5kdc[2523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:41:51 dist-test-slave-zr1t krb5kdc[2523](info): set up 2 sockets
Aug 06 22:41:51 dist-test-slave-zr1t krb5kdc[2523](info): commencing operation
krb5kdc: starting...
W20260806 22:41:52.324745 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:41:52 dist-test-slave-zr1t krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:41:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.190.keytab.
Entry for principal oryx/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.190.keytab.
I20260806 22:41:52.607563 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:35307
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.190.keytab
--principal=oryx/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:35307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:52.948922 2542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:52.949265 2542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:52.949424 2542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:52.956972 2542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:52.957098 2542 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 22:41:52.957173 2542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:52.957269 2542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:52.957361 2542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:52.957482 2542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:52.969252 2542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:35307
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.190.keytab
--principal=oryx/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:35307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:52.971086 2542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:52.972806 2542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:52.985462 2548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:52.985435 2550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:52.985775 2547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:52.986773 2542 server_base.cc:1034] running on GCE node
I20260806 22:41:52.987660 2542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:52.989166 2542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:52.990447 2542 hybrid_clock.cc:648] HybridClock initialized: now 1786056112990377 us; error 45 us; skew 500 ppm
I20260806 22:41:52.990885 2542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:52 dist-test-slave-zr1t krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056112, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:52.994311 2542 init.cc:377] Logged in from keytab as oryx/127.23.100.190@KRBTEST.COM (short username oryx)
I20260806 22:41:52.995915 2542 webserver.cc:460] Webserver started at http://127.23.100.190:38039/ using document root <none> and password file <none>
I20260806 22:41:52.996802 2542 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:52.996953 2542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:52.997344 2542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:53.000440 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "dee621521a974961bbd1844e8249b0e9"
format_stamp: "Formatted at 2026-08-06 22:41:52 on dist-test-slave-zr1t"
server_key: "bdca6e3614b36902945e94a441af2b50"
server_key_iv: "781671736d2880bdef8cf4d3c2953a29"
server_key_version: "encryptionkey@0"
I20260806 22:41:53.001292 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "dee621521a974961bbd1844e8249b0e9"
format_stamp: "Formatted at 2026-08-06 22:41:52 on dist-test-slave-zr1t"
server_key: "bdca6e3614b36902945e94a441af2b50"
server_key_iv: "781671736d2880bdef8cf4d3c2953a29"
server_key_version: "encryptionkey@0"
I20260806 22:41:53.006923 2542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 22:41:53.010872 2561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:53.012245 2542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:41:53.012481 2542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "dee621521a974961bbd1844e8249b0e9"
format_stamp: "Formatted at 2026-08-06 22:41:52 on dist-test-slave-zr1t"
server_key: "bdca6e3614b36902945e94a441af2b50"
server_key_iv: "781671736d2880bdef8cf4d3c2953a29"
server_key_version: "encryptionkey@0"
I20260806 22:41:53.012740 2542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:53.035895 2542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:53.040868 2542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:53.041195 2542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:53.067082 2542 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:35307
I20260806 22:41:53.067104 2663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:35307 every 8 connection(s)
I20260806 22:41:53.068526 2542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:41:53.072693 2664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:53.077024 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 2542
I20260806 22:41:53.077279 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:41:53.078044 23954 external_mini_cluster.cc:1428] Setting key 97e0441c3e994328be74be8e6b85017a
I20260806 22:41:53.085925 2664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9: Bootstrap starting.
I20260806 22:41:53.089967 2664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:53.091389 2664 log.cc:824] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:53.094951 2664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9: No bootstrap required, opened a new log
Aug 06 22:41:53 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:53.106294 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:53.080274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.23.100.190:35307)
0806 22:41:53.081279 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:53.081296 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:53.081368 (+ 72us) server_negotiation.cc:407] Connection header received
0806 22:41:53.084048 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:53.084079 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:53.084643 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:53.085110 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:53.086782 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:53.088900 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:53.090450 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:53.091587 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:53.091792 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:53.096821 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:53.096839 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:53.096870 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:53.096925 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:53.101167 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:53.102073 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:53.102083 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:53.102094 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:53.102341 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:53.102782 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:53.102787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:53.102790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:53.103556 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:53.103899 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:53.104321 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 22:41:53.104834 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":416,"threads_started":1}
I20260806 22:41:53.106401 2664 raft_consensus.cc:348] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee621521a974961bbd1844e8249b0e9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35307 } }
I20260806 22:41:53.106878 2664 raft_consensus.cc:374] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:53.107007 2664 raft_consensus.cc:729] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee621521a974961bbd1844e8249b0e9, State: Initialized, Role: FOLLOWER
I20260806 22:41:53.107702 2664 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee621521a974961bbd1844e8249b0e9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35307 } }
I20260806 22:41:53.108011 2664 raft_consensus.cc:388] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:41:53.108165 2664 raft_consensus.cc:482] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:41:53.108378 2664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:53.110986 2664 raft_consensus.cc:504] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee621521a974961bbd1844e8249b0e9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35307 } }
I20260806 22:41:53.111572 2664 leader_election.cc:302] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dee621521a974961bbd1844e8249b0e9; no voters:
I20260806 22:41:53.112259 2664 leader_election.cc:288] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:41:53.112433 2672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:53.113833 2672 raft_consensus.cc:686] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [term 1 LEADER]: Becoming Leader. State: Replica: dee621521a974961bbd1844e8249b0e9, State: Running, Role: LEADER
I20260806 22:41:53.114408 2672 consensus_queue.cc:237] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee621521a974961bbd1844e8249b0e9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35307 } }
I20260806 22:41:53.114629 2664 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:41:53.117681 2675 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dee621521a974961bbd1844e8249b0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee621521a974961bbd1844e8249b0e9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35307 } } }
I20260806 22:41:53.117754 2676 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dee621521a974961bbd1844e8249b0e9. Latest consensus state: current_term: 1 leader_uuid: "dee621521a974961bbd1844e8249b0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee621521a974961bbd1844e8249b0e9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35307 } } }
I20260806 22:41:53.118144 2675 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:53.118155 2676 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9 [sys.catalog]: This master's current role is: LEADER
I20260806 22:41:53.119045 2688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:41:53.124212 2688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:41:53.133477 2688 catalog_manager.cc:1347] Generated new cluster ID: 6c56f1f2725c4a2b8c7ffa6ecf406043
I20260806 22:41:53.133630 2688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:41:53.148629 2688 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:41:53.150660 2688 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:41:53.164662 2688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9: Generated new TSK 0
I20260806 22:41:53.165956 2688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab.
Entry for principal oryx/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab.
I20260806 22:41:53.242031 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:35307
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab
--principal=oryx/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:53.588234 2708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:53.588519 2708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:53.588646 2708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:53.596051 2708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:53.596176 2708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:53.596267 2708 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 22:41:53.596346 2708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:41:53.606184 2708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab
--principal=oryx/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.190:35307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:53.607403 2708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:53.608769 2708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:53.622946 2724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:53.622989 2720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:53.623159 2726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:53.627143 2708 server_base.cc:1034] running on GCE node
I20260806 22:41:53.627851 2708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:53.628863 2708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:53.630136 2708 hybrid_clock.cc:648] HybridClock initialized: now 1786056113630051 us; error 57 us; skew 500 ppm
I20260806 22:41:53.630501 2708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:53 dist-test-slave-zr1t krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056113, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:53.637328 2708 init.cc:377] Logged in from keytab as oryx/127.23.100.129@KRBTEST.COM (short username oryx)
I20260806 22:41:53.639132 2708 webserver.cc:460] Webserver started at http://127.23.100.129:36025/ using document root <none> and password file <none>
I20260806 22:41:53.640007 2708 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:53.640103 2708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:53.640407 2708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:53.643654 2708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "aec4647a38b544e8a20c0dd669addce3"
format_stamp: "Formatted at 2026-08-06 22:41:53 on dist-test-slave-zr1t"
server_key: "560b5c3da99e40c8bca1a68bf464f5a9"
server_key_iv: "7a8ed3374fd80568a0f8c53277fc05e5"
server_key_version: "encryptionkey@0"
I20260806 22:41:53.644481 2708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "aec4647a38b544e8a20c0dd669addce3"
format_stamp: "Formatted at 2026-08-06 22:41:53 on dist-test-slave-zr1t"
server_key: "560b5c3da99e40c8bca1a68bf464f5a9"
server_key_iv: "7a8ed3374fd80568a0f8c53277fc05e5"
server_key_version: "encryptionkey@0"
I20260806 22:41:53.650262 2708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 22:41:53.654143 2738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:53.655614 2708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:41:53.655799 2708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "aec4647a38b544e8a20c0dd669addce3"
format_stamp: "Formatted at 2026-08-06 22:41:53 on dist-test-slave-zr1t"
server_key: "560b5c3da99e40c8bca1a68bf464f5a9"
server_key_iv: "7a8ed3374fd80568a0f8c53277fc05e5"
server_key_version: "encryptionkey@0"
I20260806 22:41:53.656054 2708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:53.703994 2708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:53.708863 2708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:53.709214 2708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:53.712352 2708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:53.712491 2708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:53.712672 2708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:53.712776 2708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 22:41:53 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056113, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:53.736786 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:53.713053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:44159 (local address 127.23.100.190:35307)
0806 22:41:53.714065 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:53.714075 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:53.715187 (+ 1112us) server_negotiation.cc:407] Connection header received
0806 22:41:53.717806 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:53.717814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:53.717943 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:53.718098 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:53.720431 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:53.721632 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:53.724346 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:53.724990 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:53.725133 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:53.729610 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:53.729618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:53.729622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:53.729666 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:53.733255 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:53.734327 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:53.734334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:53.734338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:53.734480 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:53.734940 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:53.734945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:53.734949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:53.735228 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:53.735411 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:53.736152 (+ 741us) server_negotiation.cc:299] Negotiation successful
0806 22:41:53.736406 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":649,"threads_started":1}
I20260806 22:41:53.738112 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:53.713818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:35307 (local address 127.23.100.129:44159)
0806 22:41:53.715001 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:53.715052 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:53.717424 (+ 2372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:53.718499 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:53.718517 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:53.719305 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:53.720227 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:53.720249 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:53.721849 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:53.721857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:53.724135 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:53.724154 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:53.725251 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:53.725262 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:53.725571 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:53.726346 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:53.726379 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:53.729343 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:53.733416 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:53.733435 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:53.733448 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:53.734189 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:53.734598 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:53.734603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:53.734607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:53.734809 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:53.735365 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:53.735373 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:53.735752 (+ 379us) client_negotiation.cc:769] Sending connection context
0806 22:41:53.736040 (+ 288us) client_negotiation.cc:240] Negotiation successful
0806 22:41:53.736582 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"thread_start_us":560,"threads_started":1}
I20260806 22:41:53.768640 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:53.752185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:35307 (local address 127.23.100.129:53095)
0806 22:41:53.753051 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:53.753082 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:53.753312 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:53.753784 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:53.753789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:53.754554 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:53.755047 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:53.755067 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:53.756556 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:53.756560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:53.758129 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:53.758144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:53.759198 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:53.759207 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:53.759379 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:53.760167 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:53.760185 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:53.761639 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:53.764945 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:53.764963 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:53.764971 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:53.765803 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:53.766272 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:53.766277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:53.766281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:53.766501 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:53.767245 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:53.767250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:53.767480 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 22:41:53.767649 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 22:41:53.768070 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":405,"threads_started":1}
I20260806 22:41:53.769687 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:53.752235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:53095 (local address 127.23.100.190:35307)
0806 22:41:53.752506 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:53.752515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:53.753202 (+ 687us) server_negotiation.cc:407] Connection header received
0806 22:41:53.753491 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:53.753495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:53.753631 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:53.753748 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:53.755291 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:53.756317 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:53.758296 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:53.758988 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:53.759130 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:53.761895 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:53.761900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:53.761903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:53.761936 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:53.764781 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:53.765960 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:53.765966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:53.765970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:53.766129 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:53.766672 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:53.766677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:53.766680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:53.767068 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:53.768966 (+ 1898us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:53.769137 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 22:41:53.769360 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 22:41:53.773473 2708 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:46119
I20260806 22:41:53.773500 2915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:46119 every 8 connection(s)
I20260806 22:41:53.774880 2708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:41:53.777011 2916 heartbeater.cc:347] Connected to a master server at 127.23.100.190:35307
I20260806 22:41:53.777273 2916 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:53.778059 2916 heartbeater.cc:511] Master 127.23.100.190:35307 requested a full tablet report, sending...
I20260806 22:41:53.780251 2587 ts_manager.cc:194] Registered new tserver with Master: aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119)
I20260806 22:41:53.781831 2587 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:44159
I20260806 22:41:53.782505 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 2708
I20260806 22:41:53.782728 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:41:53.783569 23954 external_mini_cluster.cc:1428] Setting key 7c21761783b46ae2968b8ca1de4edf83
WARNING: no policy specified for oryx/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab.
Entry for principal oryx/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab.
I20260806 22:41:53.846895 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:35307
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab
--principal=oryx/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:54.199288 2926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:54.199580 2926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:54.199677 2926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:54.207162 2926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:54.207294 2926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:54.207382 2926 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 22:41:54.207465 2926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:41:54.217864 2926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab
--principal=oryx/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.190:35307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:54.219233 2926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:54.220594 2926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:54.235503 2935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:54.235540 2938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:54.235503 2933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:54.236915 2926 server_base.cc:1034] running on GCE node
I20260806 22:41:54.237742 2926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:54.238804 2926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:54.240135 2926 hybrid_clock.cc:648] HybridClock initialized: now 1786056114240049 us; error 58 us; skew 500 ppm
I20260806 22:41:54.240593 2926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:54 dist-test-slave-zr1t krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056114, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:54.244619 2926 init.cc:377] Logged in from keytab as oryx/127.23.100.130@KRBTEST.COM (short username oryx)
I20260806 22:41:54.246524 2926 webserver.cc:460] Webserver started at http://127.23.100.130:36473/ using document root <none> and password file <none>
I20260806 22:41:54.247326 2926 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:54.247479 2926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:54.247854 2926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:54.251286 2926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "7a9e43e3397a4b5e915942b7dcad3d37"
format_stamp: "Formatted at 2026-08-06 22:41:54 on dist-test-slave-zr1t"
server_key: "d2dddb02f70b5c276589db5f9f3d8777"
server_key_iv: "802be1da1b74152f3df49273515f2809"
server_key_version: "encryptionkey@0"
I20260806 22:41:54.252310 2926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "7a9e43e3397a4b5e915942b7dcad3d37"
format_stamp: "Formatted at 2026-08-06 22:41:54 on dist-test-slave-zr1t"
server_key: "d2dddb02f70b5c276589db5f9f3d8777"
server_key_iv: "802be1da1b74152f3df49273515f2809"
server_key_version: "encryptionkey@0"
I20260806 22:41:54.258165 2926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260806 22:41:54.262140 2946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:54.263572 2926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:41:54.263774 2926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "7a9e43e3397a4b5e915942b7dcad3d37"
format_stamp: "Formatted at 2026-08-06 22:41:54 on dist-test-slave-zr1t"
server_key: "d2dddb02f70b5c276589db5f9f3d8777"
server_key_iv: "802be1da1b74152f3df49273515f2809"
server_key_version: "encryptionkey@0"
I20260806 22:41:54.264029 2926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:54.288908 2926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:54.293632 2926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:54.293928 2926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:54.296898 2926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:54.297027 2926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:54.297187 2926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:54.297283 2926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 22:41:54 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056114, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:54.324841 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:54.297794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:40247 (local address 127.23.100.190:35307)
0806 22:41:54.298353 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:54.298361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:54.299771 (+ 1410us) server_negotiation.cc:407] Connection header received
0806 22:41:54.303225 (+ 3454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:54.303232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:54.303344 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:54.303511 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:54.306109 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.307277 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.309861 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.310457 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.310600 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:54.316288 (+ 5688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:54.316299 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:54.316307 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:54.316360 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:54.320428 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:54.321859 (+ 1431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:54.321866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:54.321869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:54.322052 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:54.322703 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:54.322709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:54.322712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:54.323023 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:54.323207 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:54.324196 (+ 989us) server_negotiation.cc:299] Negotiation successful
0806 22:41:54.324442 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":307,"threads_started":1}
I20260806 22:41:54.326603 2959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:54.298579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:35307 (local address 127.23.100.130:40247)
0806 22:41:54.299555 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:54.299621 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:54.302698 (+ 3077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:54.303824 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:54.303845 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:54.304812 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:54.305856 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:54.305899 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.307464 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.307473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:54.309684 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:54.309700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.310753 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.310762 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:54.311177 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:54.312155 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:54.312197 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:54.315941 (+ 3744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:54.320655 (+ 4714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:54.320668 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:54.320693 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:54.321655 (+ 962us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:54.322187 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:54.322195 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:54.322202 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:54.322577 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:54.323180 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:54.323189 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:54.323677 (+ 488us) client_negotiation.cc:769] Sending connection context
0806 22:41:54.324084 (+ 407us) client_negotiation.cc:240] Negotiation successful
0806 22:41:54.324737 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":439,"threads_started":1}
I20260806 22:41:54.342334 2926 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:39933
I20260806 22:41:54.342396 3121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:39933 every 8 connection(s)
I20260806 22:41:54.343655 2926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:41:54.346812 3123 heartbeater.cc:347] Connected to a master server at 127.23.100.190:35307
I20260806 22:41:54.347065 3123 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:54.347470 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 2926
I20260806 22:41:54.347663 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:41:54.347970 3123 heartbeater.cc:511] Master 127.23.100.190:35307 requested a full tablet report, sending...
I20260806 22:41:54.348337 23954 external_mini_cluster.cc:1428] Setting key f8f7f128dd21760d4fa3f175b517ad5d
I20260806 22:41:54.349938 2586 ts_manager.cc:194] Registered new tserver with Master: 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933)
I20260806 22:41:54.351166 2586 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:40247
I20260806 22:41:54.352761 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:54.334211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:49305 (local address 127.23.100.190:35307)
0806 22:41:54.334400 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:54.334405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:54.335882 (+ 1477us) server_negotiation.cc:407] Connection header received
0806 22:41:54.336150 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:54.336156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:54.336255 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:54.336378 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:54.340290 (+ 3912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.341363 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.343011 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.343436 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.343580 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:54.346221 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:54.346228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:54.346233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:54.346272 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:54.349642 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:54.350470 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:54.350475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:54.350478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:54.350625 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:54.351047 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:54.351052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:54.351056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:54.351584 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:54.351740 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:54.352171 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 22:41:54.352399 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 22:41:54.353235 3103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:54.334729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:35307 (local address 127.23.100.130:49305)
0806 22:41:54.335743 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:54.335770 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:54.335959 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:54.336604 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:54.336613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:54.337545 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:54.340065 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:54.340083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.341579 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.341585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:54.342833 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:54.342847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.343646 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.343653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:54.343878 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:54.344627 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:54.344645 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:54.346001 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:54.349794 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:54.349799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:54.349803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:54.350327 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:54.350756 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:54.350762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:54.350765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:54.350918 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:54.351701 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:54.351707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:54.351895 (+ 188us) client_negotiation.cc:769] Sending connection context
0806 22:41:54.352058 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 22:41:54.352415 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":593,"threads_started":1}
WARNING: no policy specified for oryx/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab.
Entry for principal oryx/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab.
I20260806 22:41:54.413254 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:35307
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab
--principal=oryx/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:54.760880 3129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:54.761215 3129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:54.761396 3129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:54.769156 3129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:54.769310 3129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:54.769476 3129 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 22:41:54.769632 3129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:41:54.779721 3129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab
--principal=oryx/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.100.190:35307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:54.780932 3129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:54.782363 3129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 22:41:54.785101 2916 heartbeater.cc:503] Master 127.23.100.190:35307 was elected leader, sending a full tablet report...
W20260806 22:41:54.797705 3136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:54.797853 3139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:54.797741 3137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:54.798663 3129 server_base.cc:1034] running on GCE node
I20260806 22:41:54.799388 3129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:54.800423 3129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:54.801700 3129 hybrid_clock.cc:648] HybridClock initialized: now 1786056114801632 us; error 47 us; skew 500 ppm
I20260806 22:41:54.802060 3129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:41:54 dist-test-slave-zr1t krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056114, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:41:54.805755 3129 init.cc:377] Logged in from keytab as oryx/127.23.100.131@KRBTEST.COM (short username oryx)
I20260806 22:41:54.807461 3129 webserver.cc:460] Webserver started at http://127.23.100.131:34313/ using document root <none> and password file <none>
I20260806 22:41:54.808282 3129 fs_manager.cc:359] Metadata directory not provided
I20260806 22:41:54.808423 3129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:41:54.808809 3129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:41:54.812155 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "a523f5ba05294594b1fdb3d7c74e58ce"
format_stamp: "Formatted at 2026-08-06 22:41:54 on dist-test-slave-zr1t"
server_key: "5e96b6f03f94b41e8a663fa8656deaf9"
server_key_iv: "f645c9605e5640be5863c8caa815ca47"
server_key_version: "encryptionkey@0"
I20260806 22:41:54.813076 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "a523f5ba05294594b1fdb3d7c74e58ce"
format_stamp: "Formatted at 2026-08-06 22:41:54 on dist-test-slave-zr1t"
server_key: "5e96b6f03f94b41e8a663fa8656deaf9"
server_key_iv: "f645c9605e5640be5863c8caa815ca47"
server_key_version: "encryptionkey@0"
I20260806 22:41:54.819361 3129 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 22:41:54.823585 3151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:54.824954 3129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 22:41:54.825160 3129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "a523f5ba05294594b1fdb3d7c74e58ce"
format_stamp: "Formatted at 2026-08-06 22:41:54 on dist-test-slave-zr1t"
server_key: "5e96b6f03f94b41e8a663fa8656deaf9"
server_key_iv: "f645c9605e5640be5863c8caa815ca47"
server_key_version: "encryptionkey@0"
I20260806 22:41:54.825417 3129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:41:54.840481 3129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:41:54.844945 3129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:41:54.845239 3129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:41:54.848006 3129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:41:54.848120 3129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:41:54.848289 3129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:41:54.848389 3129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 22:41:54 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056114, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:54.875394 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:54.849457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:35307 (local address 127.23.100.131:33029)
0806 22:41:54.850496 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:54.850557 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:54.853048 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:54.853913 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:54.853927 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:54.854766 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:54.855729 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:54.855750 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.857253 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.857266 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:54.860370 (+ 3104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:54.860391 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.861706 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.861717 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:54.861954 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:54.862704 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:54.862728 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:54.865157 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:54.869407 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:54.869417 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:54.869438 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:54.870316 (+ 878us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:54.870853 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:54.870863 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:54.870867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:54.871196 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:54.871906 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:54.871918 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:54.872476 (+ 558us) client_negotiation.cc:769] Sending connection context
0806 22:41:54.872926 (+ 450us) client_negotiation.cc:240] Negotiation successful
0806 22:41:54.873568 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":422,"threads_started":1}
I20260806 22:41:54.876555 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:54.849019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:33029 (local address 127.23.100.190:35307)
0806 22:41:54.849255 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:54.849262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:54.850699 (+ 1437us) server_negotiation.cc:407] Connection header received
0806 22:41:54.853361 (+ 2662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:54.853368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:54.853486 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:54.853640 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:54.855932 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.857023 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.860579 (+ 3556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.861326 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.861499 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:54.865414 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:54.865419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:54.865423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:54.865458 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:54.869214 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:54.870499 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:54.870507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:54.870512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:54.870700 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:54.871400 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:54.871404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:54.871407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:54.871697 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:54.871895 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:54.876072 (+ 4177us) server_negotiation.cc:299] Negotiation successful
0806 22:41:54.876284 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 22:41:54.902850 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:54.885024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:46429 (local address 127.23.100.190:35307)
0806 22:41:54.885230 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:54.885236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:54.886162 (+ 926us) server_negotiation.cc:407] Connection header received
0806 22:41:54.886520 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:54.886526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:54.886612 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:54.886747 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:54.888653 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.889976 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.892251 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.892738 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.892891 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:54.895616 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:54.895623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:54.895628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:54.895677 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:54.899411 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:54.900435 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:54.900445 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:54.900449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:54.900606 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:54.901010 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:54.901018 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:54.901021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:54.901331 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:54.901502 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:54.902125 (+ 623us) server_negotiation.cc:299] Negotiation successful
0806 22:41:54.902401 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 22:41:54.902860 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:54.885151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:35307 (local address 127.23.100.131:46429)
0806 22:41:54.886021 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:54.886053 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:54.886311 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:54.887008 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:54.887014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:54.887831 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:54.888467 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:54.888485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.890229 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.890241 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:54.892045 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:54.892060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.892961 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.892972 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:54.893140 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:54.893969 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:54.894004 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:54.895294 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:54.899569 (+ 4275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:54.899576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:54.899584 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:54.900287 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:54.900733 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:54.900739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:54.900743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:54.900891 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:54.901465 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:54.901470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:54.901769 (+ 299us) client_negotiation.cc:769] Sending connection context
0806 22:41:54.902000 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 22:41:54.902322 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":498,"threads_started":1}
I20260806 22:41:54.904531 3129 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:39757
I20260806 22:41:54.905866 3336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:39757 every 8 connection(s)
I20260806 22:41:54.906323 3129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:41:54.908550 3339 heartbeater.cc:347] Connected to a master server at 127.23.100.190:35307
I20260806 22:41:54.908798 3339 heartbeater.cc:464] Registering TS with master...
I20260806 22:41:54.909443 3339 heartbeater.cc:511] Master 127.23.100.190:35307 requested a full tablet report, sending...
I20260806 22:41:54.910796 2586 ts_manager.cc:194] Registered new tserver with Master: a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131:39757)
I20260806 22:41:54.911631 2586 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:33029
I20260806 22:41:54.914105 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 3129
I20260806 22:41:54.914286 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:41:54.914752 23954 external_mini_cluster.cc:1428] Setting key 74bc9cda15be9e34a04c15824f47c0d3
I20260806 22:41:54.917842 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 22:41:54 dist-test-slave-zr1t krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056114, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 22:41:54 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:41:54 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 22:41:54.983771 2958 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:41:54.972271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.23.100.190:35307)
0806 22:41:54.972584 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:54.972589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:54.972741 (+ 152us) server_negotiation.cc:407] Connection header received
0806 22:41:54.972920 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:54.972924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:54.972997 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:54.973086 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:54.974751 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.975504 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.976829 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.977225 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.977321 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:54.983459 (+ 6138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42884: BlockingRecv error: recv got EOF from 127.0.0.1:42884 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 22:41:54 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056114, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:55.000648 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:54.986813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.23.100.190:35307)
0806 22:41:54.987055 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:54.987061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:54.987323 (+ 262us) server_negotiation.cc:407] Connection header received
0806 22:41:54.987566 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:54.987571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:54.987636 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:54.987741 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:54.988859 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.989656 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.990926 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:54.991378 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:54.991522 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:54.995326 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:54.995331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:54.995334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:54.995368 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:54.997962 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:54.998709 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:54.998716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:54.998719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:54.998853 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:54.999226 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:54.999232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:54.999235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:54.999523 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:54.999799 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.000105 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.000332 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 22:41:55.004799 2586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:55.011757 2586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:55.037681 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.024306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:39757 (local address 127.0.0.1:48538)
0806 22:41:55.025176 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.025216 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.025390 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.027603 (+ 2213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.027617 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.027643 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:55.028118 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.028136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.030820 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.030830 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.033844 (+ 3014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.033865 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.036453 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.036463 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.036761 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.036839 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.036982 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.037206 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":29,"thread_start_us":483,"threads_started":1}
I20260806 22:41:55.038767 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.024447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:39933 (local address 127.0.0.1:45864)
0806 22:41:55.027290 (+ 2843us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.027310 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.027429 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.028560 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.028567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.028607 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:55.029264 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.029282 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.032107 (+ 2825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.032115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.035770 (+ 3655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.035792 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.038091 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.038095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.038213 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.038235 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.038296 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.038377 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2680,"mutex_wait_us":5,"thread_start_us":2517,"threads_started":1}
I20260806 22:41:55.039003 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.024626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48538 (local address 127.23.100.131:39757)
0806 22:41:55.026816 (+ 2190us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.026822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.026848 (+ 26us) server_negotiation.cc:407] Connection header received
0806 22:41:55.026922 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.026927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.027359 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.028342 (+ 983us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:55.028680 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.030623 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.034110 (+ 3487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.036175 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.038241 (+ 2066us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.038436 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.038509 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.038675 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2046,"thread_start_us":1921,"threads_started":1}
I20260806 22:41:55.039815 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.025199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.23.100.130:39933)
0806 22:41:55.026390 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.026397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.027579 (+ 1182us) server_negotiation.cc:407] Connection header received
0806 22:41:55.027688 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.027698 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.028061 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.028288 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:55.029536 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.031879 (+ 2343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.036030 (+ 4151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.037482 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.037707 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.039285 (+ 1578us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.039361 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.039515 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1061,"thread_start_us":935,"threads_started":1}
I20260806 22:41:55.043296 3023 tablet_service.cc:1467] Processing CreateTablet for tablet 0127e5c968a7409a8a6b64832a06fd67 (DEFAULT_TABLE table=test-table [id=05bd541bc08f4cc69887f8eaf06680ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:55.043864 3229 tablet_service.cc:1467] Processing CreateTablet for tablet 0127e5c968a7409a8a6b64832a06fd67 (DEFAULT_TABLE table=test-table [id=05bd541bc08f4cc69887f8eaf06680ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:55.045147 3023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0127e5c968a7409a8a6b64832a06fd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.045195 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.027371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:46119 (local address 127.0.0.1:47984)
0806 22:41:55.037274 (+ 9903us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.037291 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.037405 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.038568 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.038573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.038605 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:55.038986 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.038997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.040895 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.040903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.042458 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.042471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.044404 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.044411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.044611 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.044647 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.044721 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.044856 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9786,"thread_start_us":11347,"threads_started":1}
I20260806 22:41:55.046034 3229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0127e5c968a7409a8a6b64832a06fd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.046264 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.027940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.23.100.129:46119)
0806 22:41:55.037947 (+ 10007us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.037956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.037985 (+ 29us) server_negotiation.cc:407] Connection header received
0806 22:41:55.038094 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.038105 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.038344 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.038563 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:55.039172 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.040744 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.042618 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.044166 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.045502 (+ 1336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.045711 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.045787 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.045950 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9832,"thread_start_us":9695,"threads_started":1}
I20260806 22:41:55.053290 2815 tablet_service.cc:1467] Processing CreateTablet for tablet 0127e5c968a7409a8a6b64832a06fd67 (DEFAULT_TABLE table=test-table [id=05bd541bc08f4cc69887f8eaf06680ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:55.054888 2815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0127e5c968a7409a8a6b64832a06fd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.062443 3372 tablet_bootstrap.cc:492] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37: Bootstrap starting.
I20260806 22:41:55.063352 3371 tablet_bootstrap.cc:492] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: Bootstrap starting.
I20260806 22:41:55.067229 3371 tablet_bootstrap.cc:654] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.068257 3372 tablet_bootstrap.cc:654] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.068540 3371 log.cc:824] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:55.069747 3372 log.cc:824] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:55.073962 3374 tablet_bootstrap.cc:492] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3: Bootstrap starting.
I20260806 22:41:55.081764 3374 tablet_bootstrap.cc:654] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.082533 3371 tablet_bootstrap.cc:492] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: No bootstrap required, opened a new log
I20260806 22:41:55.082816 3371 ts_tablet_manager.cc:1397] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.008s
I20260806 22:41:55.083316 3372 tablet_bootstrap.cc:492] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37: No bootstrap required, opened a new log
I20260806 22:41:55.083506 3374 log.cc:824] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3: Log is configured to *not* fsync() on all Append() calls
I20260806 22:41:55.083621 3372 ts_tablet_manager.cc:1397] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.005s
I20260806 22:41:55.093628 3371 raft_consensus.cc:348] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } }
I20260806 22:41:55.094092 3371 raft_consensus.cc:374] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.094197 3371 raft_consensus.cc:729] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a523f5ba05294594b1fdb3d7c74e58ce, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.094835 3371 consensus_queue.cc:260] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } }
I20260806 22:41:55.095785 3374 tablet_bootstrap.cc:492] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3: No bootstrap required, opened a new log
I20260806 22:41:55.095995 3339 heartbeater.cc:503] Master 127.23.100.190:35307 was elected leader, sending a full tablet report...
I20260806 22:41:55.096097 3374 ts_tablet_manager.cc:1397] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.009s
I20260806 22:41:55.095904 3372 raft_consensus.cc:348] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } }
I20260806 22:41:55.096328 3372 raft_consensus.cc:374] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.096447 3372 raft_consensus.cc:729] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9e43e3397a4b5e915942b7dcad3d37, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.096679 3371 ts_tablet_manager.cc:1428] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 22:41:55.097122 3372 consensus_queue.cc:260] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } }
I20260806 22:41:55.098338 3123 heartbeater.cc:503] Master 127.23.100.190:35307 was elected leader, sending a full tablet report...
I20260806 22:41:55.098943 3372 ts_tablet_manager.cc:1428] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260806 22:41:55.105186 3374 raft_consensus.cc:348] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } }
I20260806 22:41:55.105568 3374 raft_consensus.cc:374] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.105665 3374 raft_consensus.cc:729] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec4647a38b544e8a20c0dd669addce3, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.106235 3374 consensus_queue.cc:260] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } }
I20260806 22:41:55.107889 3374 ts_tablet_manager.cc:1428] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260806 22:41:55.118153 3381 raft_consensus.cc:482] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:55.118522 3381 raft_consensus.cc:504] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } }
I20260806 22:41:55.120119 3381 leader_election.cc:288] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933), aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119)
I20260806 22:41:55.127108 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.120513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:39933 (local address 127.23.100.131:40125)
0806 22:41:55.120716 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.120745 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.120870 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.121365 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.121379 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.121420 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:55.121711 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.121727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.123187 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.123190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.125019 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.125035 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.126500 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.126506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.126696 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.126722 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.126800 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.126901 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 22:41:55.127326 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.120653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:40125 (local address 127.23.100.130:39933)
0806 22:41:55.120860 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.120867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.120891 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:41:55.120994 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.120999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.121079 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.121181 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:55.121887 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.123030 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.125184 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.126338 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.126510 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.126593 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.126865 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.127014 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 22:41:55.128373 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.120930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:46119 (local address 127.23.100.131:39301)
0806 22:41:55.121898 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.121913 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.122006 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.122352 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.122355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.122374 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:55.122657 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.122665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.123828 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.123838 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.125674 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.125689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.127568 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.127575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.127829 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.127876 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.127981 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.128122 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":596,"threads_started":1}
I20260806 22:41:55.128494 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.121062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:39301 (local address 127.23.100.129:46119)
0806 22:41:55.121344 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.121354 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.121998 (+ 644us) server_negotiation.cc:407] Connection header received
0806 22:41:55.122102 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.122108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.122202 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.122327 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:55.122832 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.123685 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.125849 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.127130 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.127296 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.127391 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.128056 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.128210 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 22:41:55.128602 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0127e5c968a7409a8a6b64832a06fd67" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" is_pre_election: true
I20260806 22:41:55.129204 3051 raft_consensus.cc:2454] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 0.
I20260806 22:41:55.129637 2850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0127e5c968a7409a8a6b64832a06fd67" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec4647a38b544e8a20c0dd669addce3" is_pre_election: true
I20260806 22:41:55.129876 3157 leader_election.cc:302] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9e43e3397a4b5e915942b7dcad3d37, a523f5ba05294594b1fdb3d7c74e58ce; no voters:
I20260806 22:41:55.130254 3381 raft_consensus.cc:2781] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:55.130180 2850 raft_consensus.cc:2454] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 0.
I20260806 22:41:55.130373 3381 raft_consensus.cc:482] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:55.130486 3381 raft_consensus.cc:3037] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:55.132861 3381 raft_consensus.cc:504] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } }
I20260806 22:41:55.133572 3381 leader_election.cc:288] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 election: Requested vote from peers 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933), aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119)
I20260806 22:41:55.134012 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0127e5c968a7409a8a6b64832a06fd67" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e43e3397a4b5e915942b7dcad3d37"
I20260806 22:41:55.134096 2850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0127e5c968a7409a8a6b64832a06fd67" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec4647a38b544e8a20c0dd669addce3"
I20260806 22:41:55.134205 3051 raft_consensus.cc:3037] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:55.134346 2850 raft_consensus.cc:3037] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:55.136489 3051 raft_consensus.cc:2454] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 1.
I20260806 22:41:55.136531 2850 raft_consensus.cc:2454] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 1.
I20260806 22:41:55.136931 3157 leader_election.cc:302] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9e43e3397a4b5e915942b7dcad3d37, a523f5ba05294594b1fdb3d7c74e58ce; no voters:
I20260806 22:41:55.137249 3381 raft_consensus.cc:2781] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:55.137737 3381 raft_consensus.cc:686] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 LEADER]: Becoming Leader. State: Replica: a523f5ba05294594b1fdb3d7c74e58ce, State: Running, Role: LEADER
I20260806 22:41:55.138288 3381 consensus_queue.cc:237] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } }
I20260806 22:41:55.142920 2587 catalog_manager.cc:5495] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce reported cstate change: term changed from 0 to 1, leader changed from <none> to a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131). New cstate: current_term: 1 leader_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } health_report { overall_health: UNKNOWN } } }
W20260806 22:41:55.158174 3340 tablet.cc:2369] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:41:55.164079 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.157924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.23.100.131:39757)
0806 22:41:55.158148 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.158161 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.158216 (+ 55us) server_negotiation.cc:407] Connection header received
0806 22:41:55.158539 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.158543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.158629 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.158790 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:55.159179 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.160027 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.161560 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.162159 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.162310 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.162945 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:55.163474 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:55.163641 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.163703 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.163789 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:41:55.172331 3051 raft_consensus.cc:1261] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Refusing update from remote peer a523f5ba05294594b1fdb3d7c74e58ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:55.172331 2850 raft_consensus.cc:1261] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Refusing update from remote peer a523f5ba05294594b1fdb3d7c74e58ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:55.173283 3387 consensus_queue.cc:1035] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:55.173782 3381 consensus_queue.cc:1035] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:55.200328 2587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:55.200628 2587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:55.204268 2587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9: Sending DeleteTablet for 3 replicas of tablet 0127e5c968a7409a8a6b64832a06fd67
I20260806 22:41:55.205436 3229 tablet_service.cc:1514] Processing DeleteTablet for tablet 0127e5c968a7409a8a6b64832a06fd67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:55 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:48538
I20260806 22:41:55.205574 3023 tablet_service.cc:1514] Processing DeleteTablet for tablet 0127e5c968a7409a8a6b64832a06fd67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:55 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:45864
I20260806 22:41:55.206571 2815 tablet_service.cc:1514] Processing DeleteTablet for tablet 0127e5c968a7409a8a6b64832a06fd67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:55 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:47984
I20260806 22:41:55.206600 2586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:55.207438 2586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:55.209385 3408 tablet_replica.cc:330] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37: stopping tablet replica
I20260806 22:41:55.209599 3407 tablet_replica.cc:330] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: stopping tablet replica
I20260806 22:41:55.209859 3408 raft_consensus.cc:2229] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:55.209965 3407 raft_consensus.cc:2229] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:55.210364 3408 raft_consensus.cc:2258] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:55.210659 3407 raft_consensus.cc:2258] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:55.211153 3409 tablet_replica.cc:330] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3: stopping tablet replica
I20260806 22:41:55.211551 3409 raft_consensus.cc:2229] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:55.212011 3409 raft_consensus.cc:2258] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:55.213979 3409 ts_tablet_manager.cc:1905] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:55.215653 3407 ts_tablet_manager.cc:1905] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:55.216836 3408 ts_tablet_manager.cc:1905] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:55.217667 2815 tablet_service.cc:1467] Processing CreateTablet for tablet c2f4ab2dc22c4c73884a53604a1468b7 (DEFAULT_TABLE table=test-table [id=24d5409857c74109962d3958d7691d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:55.217878 3023 tablet_service.cc:1467] Processing CreateTablet for tablet c2f4ab2dc22c4c73884a53604a1468b7 (DEFAULT_TABLE table=test-table [id=24d5409857c74109962d3958d7691d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:55.218341 3023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f4ab2dc22c4c73884a53604a1468b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.218706 2815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f4ab2dc22c4c73884a53604a1468b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.222990 3229 tablet_service.cc:1467] Processing CreateTablet for tablet c2f4ab2dc22c4c73884a53604a1468b7 (DEFAULT_TABLE table=test-table [id=24d5409857c74109962d3958d7691d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:55.223465 3229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f4ab2dc22c4c73884a53604a1468b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.225152 3374 tablet_bootstrap.cc:492] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3: Bootstrap starting.
I20260806 22:41:55.226976 3374 tablet_bootstrap.cc:654] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.228569 3372 tablet_bootstrap.cc:492] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37: Bootstrap starting.
I20260806 22:41:55.231009 3372 tablet_bootstrap.cc:654] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.233201 3372 tablet_bootstrap.cc:492] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37: No bootstrap required, opened a new log
I20260806 22:41:55.233351 3372 ts_tablet_manager.cc:1397] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260806 22:41:55.234154 3371 tablet_bootstrap.cc:492] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce: Bootstrap starting.
I20260806 22:41:55.235227 3409 ts_tablet_manager.cc:1918] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:55.235399 3409 log.cc:1196] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/0127e5c968a7409a8a6b64832a06fd67
I20260806 22:41:55.235333 3372 raft_consensus.cc:348] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.235599 3372 raft_consensus.cc:374] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.235693 3372 raft_consensus.cc:729] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9e43e3397a4b5e915942b7dcad3d37, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.235917 3371 tablet_bootstrap.cc:654] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.235973 3409 ts_tablet_manager.cc:1939] T 0127e5c968a7409a8a6b64832a06fd67 P aec4647a38b544e8a20c0dd669addce3: Deleting consensus metadata
I20260806 22:41:55.236047 3372 consensus_queue.cc:260] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.237210 3372 ts_tablet_manager.cc:1428] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260806 22:41:55.237920 2566 catalog_manager.cc:4841] TS aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119): tablet 0127e5c968a7409a8a6b64832a06fd67 (table test-table [id=05bd541bc08f4cc69887f8eaf06680ce]) successfully deleted
I20260806 22:41:55.240831 3408 ts_tablet_manager.cc:1918] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:55.241065 3408 log.cc:1196] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/0127e5c968a7409a8a6b64832a06fd67
I20260806 22:41:55.241827 3408 ts_tablet_manager.cc:1939] T 0127e5c968a7409a8a6b64832a06fd67 P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting consensus metadata
I20260806 22:41:55.244031 2566 catalog_manager.cc:4841] TS 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933): tablet 0127e5c968a7409a8a6b64832a06fd67 (table test-table [id=05bd541bc08f4cc69887f8eaf06680ce]) successfully deleted
I20260806 22:41:55.244663 3407 ts_tablet_manager.cc:1918] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:55.244895 3407 log.cc:1196] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/0127e5c968a7409a8a6b64832a06fd67
I20260806 22:41:55.246234 3407 ts_tablet_manager.cc:1939] T 0127e5c968a7409a8a6b64832a06fd67 P a523f5ba05294594b1fdb3d7c74e58ce: Deleting consensus metadata
I20260806 22:41:55.246774 3374 tablet_bootstrap.cc:492] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3: No bootstrap required, opened a new log
I20260806 22:41:55.246941 3374 ts_tablet_manager.cc:1397] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.008s
I20260806 22:41:55.247812 3374 raft_consensus.cc:348] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.248066 3374 raft_consensus.cc:374] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.248180 3374 raft_consensus.cc:729] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec4647a38b544e8a20c0dd669addce3, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.248478 2565 catalog_manager.cc:4841] TS a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131:39757): tablet 0127e5c968a7409a8a6b64832a06fd67 (table test-table [id=05bd541bc08f4cc69887f8eaf06680ce]) successfully deleted
I20260806 22:41:55.248503 3374 consensus_queue.cc:260] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.249118 3374 ts_tablet_manager.cc:1428] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260806 22:41:55.250204 3371 tablet_bootstrap.cc:492] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce: No bootstrap required, opened a new log
I20260806 22:41:55.250368 3371 ts_tablet_manager.cc:1397] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260806 22:41:55.251294 3371 raft_consensus.cc:348] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.251555 3371 raft_consensus.cc:374] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.251659 3371 raft_consensus.cc:729] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a523f5ba05294594b1fdb3d7c74e58ce, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.251933 3371 consensus_queue.cc:260] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.252689 3371 ts_tablet_manager.cc:1428] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260806 22:41:55.269012 3385 raft_consensus.cc:482] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:55.269268 3385 raft_consensus.cc:504] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.270916 3385 leader_election.cc:288] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933), a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131:39757)
I20260806 22:41:55.276589 3384 raft_consensus.cc:482] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:55.276842 3384 raft_consensus.cc:504] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
W20260806 22:41:55.277577 2918 tablet.cc:2369] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:41:55.278867 3384 leader_election.cc:288] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119), a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131:39757)
I20260806 22:41:55.280035 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.271501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:36935 (local address 127.23.100.130:39933)
0806 22:41:55.271706 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.271712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.273222 (+ 1510us) server_negotiation.cc:407] Connection header received
0806 22:41:55.273452 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.273458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.273579 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.273686 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:55.274205 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.274980 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.277161 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.278434 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.278587 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.278658 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.279532 (+ 874us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.279676 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:41:55.281034 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.271337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:52817 (local address 127.23.100.131:39757)
0806 22:41:55.271559 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.271565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.273423 (+ 1858us) server_negotiation.cc:407] Connection header received
0806 22:41:55.273776 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.273782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.273875 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.273940 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:55.274772 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.275917 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.278518 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.279847 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.279977 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.280045 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.280600 (+ 555us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.280738 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 22:41:55.281067 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.271787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:39933 (local address 127.23.100.129:36935)
0806 22:41:55.273093 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.273117 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.273296 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.273716 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.273720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.273749 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:55.274049 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.274058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.275121 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.275127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.276948 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.276965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.279138 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.279150 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.279342 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.279378 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.280428 (+ 1050us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.280572 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":139,"thread_start_us":699,"threads_started":1}
I20260806 22:41:55.281930 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.272242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:39757 (local address 127.23.100.129:52817)
0806 22:41:55.273317 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.273336 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.273608 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.274162 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.274167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.274193 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:55.274566 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.274578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.276110 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.276115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.278323 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.278341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.280227 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.280233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.280416 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.280452 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.281385 (+ 933us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.281542 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":29,"thread_start_us":710,"threads_started":1}
I20260806 22:41:55.282802 3260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f4ab2dc22c4c73884a53604a1468b7" candidate_uuid: "aec4647a38b544e8a20c0dd669addce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" is_pre_election: true
I20260806 22:41:55.283353 3260 raft_consensus.cc:2454] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aec4647a38b544e8a20c0dd669addce3 in term 0.
I20260806 22:41:55.283892 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f4ab2dc22c4c73884a53604a1468b7" candidate_uuid: "aec4647a38b544e8a20c0dd669addce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" is_pre_election: true
I20260806 22:41:55.284159 3051 raft_consensus.cc:2454] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aec4647a38b544e8a20c0dd669addce3 in term 0.
I20260806 22:41:55.284680 2742 leader_election.cc:302] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9e43e3397a4b5e915942b7dcad3d37, aec4647a38b544e8a20c0dd669addce3; no voters:
I20260806 22:41:55.285161 3385 raft_consensus.cc:2781] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:55.285290 3385 raft_consensus.cc:482] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:55.285408 3385 raft_consensus.cc:3037] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:55.288751 3385 raft_consensus.cc:504] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.289535 3385 leader_election.cc:288] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [CANDIDATE]: Term 1 election: Requested vote from peers 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933), a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131:39757)
I20260806 22:41:55.290195 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f4ab2dc22c4c73884a53604a1468b7" candidate_uuid: "aec4647a38b544e8a20c0dd669addce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e43e3397a4b5e915942b7dcad3d37"
I20260806 22:41:55.290361 3260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f4ab2dc22c4c73884a53604a1468b7" candidate_uuid: "aec4647a38b544e8a20c0dd669addce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a523f5ba05294594b1fdb3d7c74e58ce"
I20260806 22:41:55.290436 3051 raft_consensus.cc:3037] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:55.290576 3260 raft_consensus.cc:3037] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:55.292085 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.280475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:58775 (local address 127.23.100.131:39757)
0806 22:41:55.281283 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.281288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.283529 (+ 2241us) server_negotiation.cc:407] Connection header received
0806 22:41:55.283913 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.283921 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.284019 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.284092 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:55.285625 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.286732 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.289271 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.290699 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.290848 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.290941 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.291571 (+ 630us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.291729 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"threads_started":1}
I20260806 22:41:55.292788 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.279943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:39757 (local address 127.23.100.130:58775)
0806 22:41:55.283420 (+ 3477us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.283441 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.283718 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.284461 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.284467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.284510 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:55.285181 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.285200 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.286938 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.286943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.289072 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.289087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.291158 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.291166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.291356 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.291397 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.292233 (+ 836us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.292377 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3341,"thread_start_us":1770,"threads_started":1}
I20260806 22:41:55.293646 3051 raft_consensus.cc:2454] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec4647a38b544e8a20c0dd669addce3 in term 1.
I20260806 22:41:55.294067 3260 raft_consensus.cc:2454] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec4647a38b544e8a20c0dd669addce3 in term 1.
I20260806 22:41:55.294013 3256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f4ab2dc22c4c73884a53604a1468b7" candidate_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" is_pre_election: true
I20260806 22:41:55.294492 3256 raft_consensus.cc:2379] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a9e43e3397a4b5e915942b7dcad3d37 in current term 1: Already voted for candidate aec4647a38b544e8a20c0dd669addce3 in this term.
I20260806 22:41:55.294543 2741 leader_election.cc:302] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a523f5ba05294594b1fdb3d7c74e58ce, aec4647a38b544e8a20c0dd669addce3; no voters:
I20260806 22:41:55.295455 3385 raft_consensus.cc:2781] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:55.295951 3385 raft_consensus.cc:686] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 1 LEADER]: Becoming Leader. State: Replica: aec4647a38b544e8a20c0dd669addce3, State: Running, Role: LEADER
I20260806 22:41:55.296416 3385 consensus_queue.cc:237] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.301002 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.282260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:50563 (local address 127.23.100.129:46119)
0806 22:41:55.282565 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.282570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.282588 (+ 18us) server_negotiation.cc:407] Connection header received
0806 22:41:55.282644 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.282649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.282718 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.282804 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:41:55.294757 (+ 11953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.295935 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.298439 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.299980 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.300152 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.300228 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.300606 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.300735 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260806 22:41:55.301105 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.279433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:46119 (local address 127.23.100.130:50563)
0806 22:41:55.282218 (+ 2785us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.282237 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.282401 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.293924 (+ 11523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.293931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.293961 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:41:55.294432 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.294448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.296137 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.296143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.298244 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.298267 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.300232 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.300239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.300421 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.300446 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.300559 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.300708 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2437,"thread_start_us":3655,"threads_started":1}
I20260806 22:41:55.301082 2587 catalog_manager.cc:5495] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aec4647a38b544e8a20c0dd669addce3 (127.23.100.129). New cstate: current_term: 1 leader_uuid: "aec4647a38b544e8a20c0dd669addce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } health_report { overall_health: UNKNOWN } } }
I20260806 22:41:55.301807 2850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f4ab2dc22c4c73884a53604a1468b7" candidate_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec4647a38b544e8a20c0dd669addce3" is_pre_election: true
I20260806 22:41:55.302708 2951 leader_election.cc:302] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a9e43e3397a4b5e915942b7dcad3d37; no voters: a523f5ba05294594b1fdb3d7c74e58ce, aec4647a38b544e8a20c0dd669addce3
I20260806 22:41:55.303530 3384 raft_consensus.cc:2726] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 22:41:55.347692 3124 tablet.cc:2369] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 22:41:55 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056112, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.190@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:55.365818 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.348126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:35307 (local address 127.0.0.1:42890)
0806 22:41:55.349030 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.349051 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.349215 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.349597 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.349602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.350509 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:41:55.350887 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.350899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.352299 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.352305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.353906 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.353920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.354956 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.354964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.355142 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.355935 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:41:55.355980 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:41:55.358916 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:41:55.362946 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:55.362960 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:55.362968 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:55.363548 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:55.364001 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:41:55.364006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:41:55.364010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:41:55.364171 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:41:55.364833 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:41:55.364843 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:41:55.365050 (+ 207us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.365185 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.365395 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":590,"threads_started":1}
I20260806 22:41:55.366076 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.348072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.23.100.190:35307)
0806 22:41:55.348295 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.348303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.349155 (+ 852us) server_negotiation.cc:407] Connection header received
0806 22:41:55.349338 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.349344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.349421 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.349538 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:55.351063 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.352155 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.354078 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.354745 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.354887 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.359118 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:55.359125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:55.359130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:55.359169 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:55.362805 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:55.363690 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:55.363695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:55.363699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:55.363880 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:55.364293 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:55.364298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:55.364302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:55.364683 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:55.364829 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.365344 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.365620 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 22:41:55.371469 2587 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:42890:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 22:41:55.372566 2587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:55.379666 2815 tablet_service.cc:1467] Processing CreateTablet for tablet fdd107584f804432844898a4a8d9dcd7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec094c290c5d4955a0c0c9ec3a9eacb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 22:41:55.380075 3023 tablet_service.cc:1467] Processing CreateTablet for tablet fdd107584f804432844898a4a8d9dcd7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec094c290c5d4955a0c0c9ec3a9eacb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 22:41:55.380280 2815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd107584f804432844898a4a8d9dcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.380636 3023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd107584f804432844898a4a8d9dcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.384305 3229 tablet_service.cc:1467] Processing CreateTablet for tablet fdd107584f804432844898a4a8d9dcd7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec094c290c5d4955a0c0c9ec3a9eacb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 22:41:55.384927 3229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd107584f804432844898a4a8d9dcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.386458 3374 tablet_bootstrap.cc:492] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3: Bootstrap starting.
I20260806 22:41:55.388165 3372 tablet_bootstrap.cc:492] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37: Bootstrap starting.
I20260806 22:41:55.388242 3374 tablet_bootstrap.cc:654] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.389936 3372 tablet_bootstrap.cc:654] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.393355 3374 tablet_bootstrap.cc:492] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3: No bootstrap required, opened a new log
I20260806 22:41:55.393594 3374 ts_tablet_manager.cc:1397] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260806 22:41:55.394224 3374 raft_consensus.cc:348] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.394505 3374 raft_consensus.cc:374] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.394590 3374 raft_consensus.cc:729] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec4647a38b544e8a20c0dd669addce3, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.394650 3371 tablet_bootstrap.cc:492] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: Bootstrap starting.
I20260806 22:41:55.394886 3374 consensus_queue.cc:260] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.395308 3385 tablet_replica.cc:439] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } }
I20260806 22:41:55.395624 3385 tablet_replica.cc:442] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:55.396564 3371 tablet_bootstrap.cc:654] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.396667 3374 ts_tablet_manager.cc:1428] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 22:41:55.401571 3371 tablet_bootstrap.cc:492] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: No bootstrap required, opened a new log
I20260806 22:41:55.401841 3371 ts_tablet_manager.cc:1397] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260806 22:41:55.402799 3371 raft_consensus.cc:348] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.403151 3371 raft_consensus.cc:374] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.403299 3371 raft_consensus.cc:729] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a523f5ba05294594b1fdb3d7c74e58ce, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.403650 3371 consensus_queue.cc:260] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.404211 3371 ts_tablet_manager.cc:1428] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 22:41:55.404091 3387 tablet_replica.cc:439] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } }
I20260806 22:41:55.404470 3387 tablet_replica.cc:442] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:55.404946 3372 tablet_bootstrap.cc:492] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37: No bootstrap required, opened a new log
I20260806 22:41:55.405148 3372 ts_tablet_manager.cc:1397] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260806 22:41:55.406061 3372 raft_consensus.cc:348] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.406322 3372 raft_consensus.cc:374] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.406424 3372 raft_consensus.cc:729] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9e43e3397a4b5e915942b7dcad3d37, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.406687 3372 consensus_queue.cc:260] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.407163 3384 tablet_replica.cc:439] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } }
I20260806 22:41:55.407469 3384 tablet_replica.cc:442] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:55.407930 3372 ts_tablet_manager.cc:1428] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 22:41:55.470209 3387 raft_consensus.cc:482] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:55.470485 3387 raft_consensus.cc:504] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.471324 3387 leader_election.cc:288] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119), 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933)
I20260806 22:41:55.471959 2850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd107584f804432844898a4a8d9dcd7" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec4647a38b544e8a20c0dd669addce3" is_pre_election: true
I20260806 22:41:55.471993 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd107584f804432844898a4a8d9dcd7" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" is_pre_election: true
I20260806 22:41:55.472258 2850 raft_consensus.cc:2454] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 0.
I20260806 22:41:55.472285 3051 raft_consensus.cc:2454] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 0.
I20260806 22:41:55.472769 3157 leader_election.cc:302] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9e43e3397a4b5e915942b7dcad3d37, a523f5ba05294594b1fdb3d7c74e58ce; no voters:
I20260806 22:41:55.473112 3387 raft_consensus.cc:2781] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:55.473232 3387 raft_consensus.cc:482] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:55.473379 3387 raft_consensus.cc:3037] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:55.476096 3387 raft_consensus.cc:504] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.476758 3387 leader_election.cc:288] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 election: Requested vote from peers aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119), 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933)
I20260806 22:41:55.477236 2850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd107584f804432844898a4a8d9dcd7" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec4647a38b544e8a20c0dd669addce3"
I20260806 22:41:55.477344 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd107584f804432844898a4a8d9dcd7" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e43e3397a4b5e915942b7dcad3d37"
I20260806 22:41:55.477512 2850 raft_consensus.cc:3037] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:55.477612 3051 raft_consensus.cc:3037] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:55.479902 2850 raft_consensus.cc:2454] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 1.
I20260806 22:41:55.480367 3157 leader_election.cc:302] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a523f5ba05294594b1fdb3d7c74e58ce, aec4647a38b544e8a20c0dd669addce3; no voters:
I20260806 22:41:55.480660 3387 raft_consensus.cc:2781] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:55.480732 3051 raft_consensus.cc:2454] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 1.
I20260806 22:41:55.480865 3387 raft_consensus.cc:686] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 LEADER]: Becoming Leader. State: Replica: a523f5ba05294594b1fdb3d7c74e58ce, State: Running, Role: LEADER
I20260806 22:41:55.481156 3387 consensus_queue.cc:237] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.483057 3381 tablet_replica.cc:439] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: TxnStatusTablet state changed. Reason: New leader a523f5ba05294594b1fdb3d7c74e58ce. Latest consensus state: current_term: 1 leader_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } }
I20260806 22:41:55.483331 3381 tablet_replica.cc:442] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: This TxnStatusTablet replica's current role is: LEADER
I20260806 22:41:55.484136 3445 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 22:41:55.484438 3445 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 22:41:55.484732 2586 catalog_manager.cc:5495] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce reported cstate change: term changed from 0 to 1, leader changed from <none> to a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131). New cstate: current_term: 1 leader_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } health_report { overall_health: HEALTHY } } }
I20260806 22:41:55.508432 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.502234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:39757 (local address 127.0.0.1:48550)
0806 22:41:55.502487 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.502510 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.502651 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.503078 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.503082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.503092 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 22:41:55.503255 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.503262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.504682 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.504687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.505957 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.505972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.507086 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.507091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.507266 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.507336 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 22:41:55.507843 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 22:41:55.507862 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.507985 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.508160 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 22:41:55.508455 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.502256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.23.100.131:39757)
0806 22:41:55.502466 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.502474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.502598 (+ 124us) server_negotiation.cc:407] Connection header received
0806 22:41:55.502823 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.502828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.502926 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.503045 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:55.503392 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.504514 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.506122 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.506890 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.507058 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.507544 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:55.507720 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:55.507823 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.508008 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.508098 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 22:41:55.512784 3051 raft_consensus.cc:1261] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Refusing update from remote peer a523f5ba05294594b1fdb3d7c74e58ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:55.512876 2850 raft_consensus.cc:1261] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Refusing update from remote peer a523f5ba05294594b1fdb3d7c74e58ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:55.513607 3387 consensus_queue.cc:1035] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:55.513983 3381 consensus_queue.cc:1035] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:55.517277 3385 tablet_replica.cc:439] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3: TxnStatusTablet state changed. Reason: New leader a523f5ba05294594b1fdb3d7c74e58ce. Latest consensus state: current_term: 1 leader_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } }
I20260806 22:41:55.517637 3385 tablet_replica.cc:442] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:55.520476 3390 mvcc.cc:205] Tried to move back new op lower bound from 7315685849133375488 to 7315685849013563392. Current Snapshot: MvccSnapshot[applied={T|T < 7315685849133375488 or (T in {7315685849133375488})}]
I20260806 22:41:55.523522 3387 tablet_replica.cc:439] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } }
I20260806 22:41:55.524009 3387 tablet_replica.cc:442] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: This TxnStatusTablet replica's current role is: LEADER
I20260806 22:41:55.528512 3384 tablet_replica.cc:439] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37: TxnStatusTablet state changed. Reason: New leader a523f5ba05294594b1fdb3d7c74e58ce. Latest consensus state: current_term: 1 leader_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } }
I20260806 22:41:55.528877 3384 tablet_replica.cc:442] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:55.531018 3385 tablet_replica.cc:439] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } }
I20260806 22:41:55.531353 3385 tablet_replica.cc:442] T fdd107584f804432844898a4a8d9dcd7 P aec4647a38b544e8a20c0dd669addce3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:55.532413 3396 tablet_replica.cc:439] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } }
I20260806 22:41:55.532867 3396 tablet_replica.cc:442] T fdd107584f804432844898a4a8d9dcd7 P a523f5ba05294594b1fdb3d7c74e58ce: This TxnStatusTablet replica's current role is: LEADER
I20260806 22:41:55.534104 3391 mvcc.cc:205] Tried to move back new op lower bound from 7315685849133375488 to 7315685849013563392. Current Snapshot: MvccSnapshot[applied={T|T < 7315685849133375488}]
I20260806 22:41:55.535316 3384 tablet_replica.cc:439] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } } }
I20260806 22:41:55.535590 3384 tablet_replica.cc:442] T fdd107584f804432844898a4a8d9dcd7 P 7a9e43e3397a4b5e915942b7dcad3d37: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 22:41:55.537581 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.529313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47986 (local address 127.23.100.129:46119)
0806 22:41:55.529561 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.529569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.529591 (+ 22us) server_negotiation.cc:407] Connection header received
0806 22:41:55.529705 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.529713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.529877 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.530068 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:55.530569 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.531826 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.534023 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.534717 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.534852 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.535582 (+ 730us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:55.536486 (+ 904us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:55.536639 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.537163 (+ 524us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.537252 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 22:41:55.554675 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.548854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:39757 (local address 127.23.100.129:59745)
0806 22:41:55.549450 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.549472 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.549679 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.550034 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.550039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.550052 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 22:41:55.550357 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.550365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.551442 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.551449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.552561 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.552573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.553356 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.553363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.553512 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.553630 (+ 118us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 22:41:55.554192 (+ 562us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 22:41:55.554202 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.554266 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.554375 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":329,"threads_started":1}
I20260806 22:41:55.554733 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.548855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:59745 (local address 127.23.100.131:39757)
0806 22:41:55.549037 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.549042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.549585 (+ 543us) server_negotiation.cc:407] Connection header received
0806 22:41:55.549818 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.549824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.549908 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.550015 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:55.550535 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.551302 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.552713 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.553170 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.553321 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.553828 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:55.554019 (+ 191us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:55.554136 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.554342 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.554443 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 22:41:55.565026 3051 raft_consensus.cc:1261] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Refusing update from remote peer aec4647a38b544e8a20c0dd669addce3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:55.565171 3256 raft_consensus.cc:1261] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Refusing update from remote peer aec4647a38b544e8a20c0dd669addce3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:55.565868 3385 consensus_queue.cc:1035] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:55.566504 3385 consensus_queue.cc:1035] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:55.602504 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.595662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:43347 (local address 127.23.100.129:46119)
0806 22:41:55.595888 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.595895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.596629 (+ 734us) server_negotiation.cc:407] Connection header received
0806 22:41:55.596861 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.596866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.596983 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.597081 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:55.597850 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.598657 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.600462 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.600948 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.601085 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.601713 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:55.601816 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:55.601930 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.602154 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.602241 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 22:41:55.603437 3459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.595742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:46119 (local address 127.23.100.131:43347)
0806 22:41:55.596517 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:55.596542 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:55.596716 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:55.597131 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:55.597136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:55.597151 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 22:41:55.597645 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.597660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.598800 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.598806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.600299 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:55.600313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.601144 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.601155 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:55.601292 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.601352 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 22:41:55.601987 (+ 635us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 22:41:55.602003 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 22:41:55.602939 (+ 936us) client_negotiation.cc:240] Negotiation successful
0806 22:41:55.603095 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":419,"threads_started":1}
I20260806 22:41:55.646637 2586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:55.646907 2586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:55.650657 2586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9: Sending DeleteTablet for 3 replicas of tablet c2f4ab2dc22c4c73884a53604a1468b7
I20260806 22:41:55.651592 3023 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f4ab2dc22c4c73884a53604a1468b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:55 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:45864
I20260806 22:41:55.651793 3229 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f4ab2dc22c4c73884a53604a1468b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:55 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:48538
I20260806 22:41:55.651903 3408 tablet_replica.cc:330] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37: stopping tablet replica
I20260806 22:41:55.652096 3407 tablet_replica.cc:330] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce: stopping tablet replica
I20260806 22:41:55.652118 3408 raft_consensus.cc:2229] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:55.652293 3407 raft_consensus.cc:2229] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:55.652344 3408 raft_consensus.cc:2258] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:55.652530 3407 raft_consensus.cc:2258] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:55.653024 2815 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f4ab2dc22c4c73884a53604a1468b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:55 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:47984
I20260806 22:41:55.653302 3409 tablet_replica.cc:330] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3: stopping tablet replica
I20260806 22:41:55.653555 3409 raft_consensus.cc:2229] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:55.653858 3407 ts_tablet_manager.cc:1905] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:55.654100 3409 raft_consensus.cc:2258] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:55.655182 3408 ts_tablet_manager.cc:1905] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:55.660775 3409 ts_tablet_manager.cc:1905] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:55.661705 3407 ts_tablet_manager.cc:1918] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:55.661900 3407 log.cc:1196] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/c2f4ab2dc22c4c73884a53604a1468b7
I20260806 22:41:55.662392 3407 ts_tablet_manager.cc:1939] T c2f4ab2dc22c4c73884a53604a1468b7 P a523f5ba05294594b1fdb3d7c74e58ce: Deleting consensus metadata
I20260806 22:41:55.663534 2565 catalog_manager.cc:4841] TS a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131:39757): tablet c2f4ab2dc22c4c73884a53604a1468b7 (table test-table [id=24d5409857c74109962d3958d7691d0a]) successfully deleted
I20260806 22:41:55.674980 3408 ts_tablet_manager.cc:1918] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:55.675321 3408 log.cc:1196] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/c2f4ab2dc22c4c73884a53604a1468b7
I20260806 22:41:55.675918 3408 ts_tablet_manager.cc:1939] T c2f4ab2dc22c4c73884a53604a1468b7 P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting consensus metadata
I20260806 22:41:55.677273 2566 catalog_manager.cc:4841] TS 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933): tablet c2f4ab2dc22c4c73884a53604a1468b7 (table test-table [id=24d5409857c74109962d3958d7691d0a]) successfully deleted
I20260806 22:41:55.679728 3409 ts_tablet_manager.cc:1918] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:55.679901 3409 log.cc:1196] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/c2f4ab2dc22c4c73884a53604a1468b7
I20260806 22:41:55.680394 3409 ts_tablet_manager.cc:1939] T c2f4ab2dc22c4c73884a53604a1468b7 P aec4647a38b544e8a20c0dd669addce3: Deleting consensus metadata
I20260806 22:41:55.681576 2566 catalog_manager.cc:4841] TS aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119): tablet c2f4ab2dc22c4c73884a53604a1468b7 (table test-table [id=24d5409857c74109962d3958d7691d0a]) successfully deleted
Aug 06 22:41:55 dist-test-slave-zr1t krb5kdc[2523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 22:41:55 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:41:55 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 22:41:55.752494 2958 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:41:55.739042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.23.100.190:35307)
0806 22:41:55.739245 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.739251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.739825 (+ 574us) server_negotiation.cc:407] Connection header received
0806 22:41:55.739999 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.740005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.740134 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.740259 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:55.742124 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.743136 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.745126 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.745818 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.745949 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.752204 (+ 6255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42904: BlockingRecv error: recv got EOF from 127.0.0.1:42904 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 22:41:55 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM
I20260806 22:41:55.776247 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:55.756384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.23.100.190:35307)
0806 22:41:55.756559 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:55.756565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:55.757013 (+ 448us) server_negotiation.cc:407] Connection header received
0806 22:41:55.757178 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:55.757184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:55.757279 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:55.757388 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:55.759080 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.760275 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.762340 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:55.762998 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:55.763137 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:55.768197 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:55.768208 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:55.768214 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:55.768339 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:55.771976 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:55.772966 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:55.772972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:55.772976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:55.773141 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:55.773701 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:55.773709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:55.773712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:55.774131 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:55.774404 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:55.775608 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0806 22:41:55.775862 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 22:41:55.780295 2586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:55.781036 2586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:55.789507 3229 tablet_service.cc:1467] Processing CreateTablet for tablet 19222552e9af4291849ba8ecd84d61ea (DEFAULT_TABLE table=test-table [id=936ddf08fd8b4ebeb71a91fe986f8f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:55.790045 3229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19222552e9af4291849ba8ecd84d61ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.797161 3371 tablet_bootstrap.cc:492] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce: Bootstrap starting.
I20260806 22:41:55.798884 2815 tablet_service.cc:1467] Processing CreateTablet for tablet 19222552e9af4291849ba8ecd84d61ea (DEFAULT_TABLE table=test-table [id=936ddf08fd8b4ebeb71a91fe986f8f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:55.799023 3371 tablet_bootstrap.cc:654] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.799378 2815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19222552e9af4291849ba8ecd84d61ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.802026 3023 tablet_service.cc:1467] Processing CreateTablet for tablet 19222552e9af4291849ba8ecd84d61ea (DEFAULT_TABLE table=test-table [id=936ddf08fd8b4ebeb71a91fe986f8f73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:55.802611 3023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19222552e9af4291849ba8ecd84d61ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:55.813477 3374 tablet_bootstrap.cc:492] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3: Bootstrap starting.
I20260806 22:41:55.815240 3374 tablet_bootstrap.cc:654] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.822288 3372 tablet_bootstrap.cc:492] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37: Bootstrap starting.
I20260806 22:41:55.824047 3372 tablet_bootstrap.cc:654] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:55.827402 3374 tablet_bootstrap.cc:492] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3: No bootstrap required, opened a new log
I20260806 22:41:55.827596 3374 ts_tablet_manager.cc:1397] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20260806 22:41:55.828689 3372 tablet_bootstrap.cc:492] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37: No bootstrap required, opened a new log
I20260806 22:41:55.828529 3374 raft_consensus.cc:348] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.828824 3374 raft_consensus.cc:374] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.828938 3374 raft_consensus.cc:729] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec4647a38b544e8a20c0dd669addce3, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.829140 3372 ts_tablet_manager.cc:1397] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260806 22:41:55.829221 3374 consensus_queue.cc:260] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.830538 3374 ts_tablet_manager.cc:1428] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260806 22:41:55.832086 3371 tablet_bootstrap.cc:492] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce: No bootstrap required, opened a new log
I20260806 22:41:55.832248 3371 ts_tablet_manager.cc:1397] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.001s
I20260806 22:41:55.833145 3371 raft_consensus.cc:348] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.833405 3371 raft_consensus.cc:374] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.833541 3371 raft_consensus.cc:729] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a523f5ba05294594b1fdb3d7c74e58ce, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.833798 3371 consensus_queue.cc:260] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.835057 3371 ts_tablet_manager.cc:1428] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 22:41:55.836131 3372 raft_consensus.cc:348] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.836439 3372 raft_consensus.cc:374] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:55.836546 3372 raft_consensus.cc:729] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9e43e3397a4b5e915942b7dcad3d37, State: Initialized, Role: FOLLOWER
I20260806 22:41:55.836815 3372 consensus_queue.cc:260] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:55.837378 3372 ts_tablet_manager.cc:1428] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20260806 22:41:56.045923 3385 raft_consensus.cc:482] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:56.046200 3385 raft_consensus.cc:504] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.047034 3385 leader_election.cc:288] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933), a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131:39757)
I20260806 22:41:56.047706 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19222552e9af4291849ba8ecd84d61ea" candidate_uuid: "aec4647a38b544e8a20c0dd669addce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" is_pre_election: true
I20260806 22:41:56.048005 3051 raft_consensus.cc:2454] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aec4647a38b544e8a20c0dd669addce3 in term 0.
I20260806 22:41:56.047904 3256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19222552e9af4291849ba8ecd84d61ea" candidate_uuid: "aec4647a38b544e8a20c0dd669addce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" is_pre_election: true
I20260806 22:41:56.048239 3256 raft_consensus.cc:2454] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aec4647a38b544e8a20c0dd669addce3 in term 0.
I20260806 22:41:56.048511 2742 leader_election.cc:302] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9e43e3397a4b5e915942b7dcad3d37, aec4647a38b544e8a20c0dd669addce3; no voters:
I20260806 22:41:56.049108 3385 raft_consensus.cc:2781] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:56.049217 3385 raft_consensus.cc:482] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:56.049328 3385 raft_consensus.cc:3037] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:56.052629 3385 raft_consensus.cc:504] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.053960 3256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19222552e9af4291849ba8ecd84d61ea" candidate_uuid: "aec4647a38b544e8a20c0dd669addce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a523f5ba05294594b1fdb3d7c74e58ce"
I20260806 22:41:56.053926 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19222552e9af4291849ba8ecd84d61ea" candidate_uuid: "aec4647a38b544e8a20c0dd669addce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e43e3397a4b5e915942b7dcad3d37"
I20260806 22:41:56.054219 3051 raft_consensus.cc:3037] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:56.054229 3256 raft_consensus.cc:3037] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:56.057404 3051 raft_consensus.cc:2454] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec4647a38b544e8a20c0dd669addce3 in term 1.
I20260806 22:41:56.057646 3256 raft_consensus.cc:2454] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec4647a38b544e8a20c0dd669addce3 in term 1.
I20260806 22:41:56.058097 2742 leader_election.cc:302] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a9e43e3397a4b5e915942b7dcad3d37, aec4647a38b544e8a20c0dd669addce3; no voters:
I20260806 22:41:56.058701 3462 raft_consensus.cc:2781] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:56.058794 3385 leader_election.cc:288] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [CANDIDATE]: Term 1 election: Requested vote from peers 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933), a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131:39757)
I20260806 22:41:56.058911 3462 raft_consensus.cc:686] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 1 LEADER]: Becoming Leader. State: Replica: aec4647a38b544e8a20c0dd669addce3, State: Running, Role: LEADER
I20260806 22:41:56.059271 3462 consensus_queue.cc:237] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.080090 2586 catalog_manager.cc:5495] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aec4647a38b544e8a20c0dd669addce3 (127.23.100.129). New cstate: current_term: 1 leader_uuid: "aec4647a38b544e8a20c0dd669addce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } health_report { overall_health: UNKNOWN } } }
I20260806 22:41:56.137413 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:56.128268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48000 (local address 127.23.100.129:46119)
0806 22:41:56.131822 (+ 3554us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:56.131826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:56.131846 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:41:56.131913 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:56.131917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:56.131982 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:56.132109 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:56.132555 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.133548 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.135241 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.135756 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.136379 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:56.136551 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:56.136676 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:56.136782 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:56.136959 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 22:41:56.137048 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3451,"thread_start_us":3284,"threads_started":1}
I20260806 22:41:56.140686 3256 raft_consensus.cc:1261] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Refusing update from remote peer aec4647a38b544e8a20c0dd669addce3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:56.141031 3051 raft_consensus.cc:1261] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Refusing update from remote peer aec4647a38b544e8a20c0dd669addce3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:56.141678 3462 consensus_queue.cc:1035] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:56.142258 3423 consensus_queue.cc:1035] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:56.167091 3491 mvcc.cc:205] Tried to move back new op lower bound from 7315685851705733120 to 7315685851382026240. Current Snapshot: MvccSnapshot[applied={T|T < 7315685851705733120}]
I20260806 22:41:56.188310 2586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:56.188584 2586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:56.192567 2586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9: Sending DeleteTablet for 3 replicas of tablet 19222552e9af4291849ba8ecd84d61ea
I20260806 22:41:56.193879 3229 tablet_service.cc:1514] Processing DeleteTablet for tablet 19222552e9af4291849ba8ecd84d61ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:56 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:48538
I20260806 22:41:56.194383 2815 tablet_service.cc:1514] Processing DeleteTablet for tablet 19222552e9af4291849ba8ecd84d61ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:56 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:47984
I20260806 22:41:56.194672 3023 tablet_service.cc:1514] Processing DeleteTablet for tablet 19222552e9af4291849ba8ecd84d61ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:56 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:45864
I20260806 22:41:56.194819 2812 tablet_service.cc:1514] Processing DeleteTablet for tablet 19222552e9af4291849ba8ecd84d61ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:56 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:47984
I20260806 22:41:56.195297 3506 tablet_replica.cc:330] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37: stopping tablet replica
I20260806 22:41:56.195582 3506 raft_consensus.cc:2229] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:56.195945 3506 raft_consensus.cc:2258] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:56.197238 3506 ts_tablet_manager.cc:1905] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:56.198055 2586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:41:56.198752 2586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:41:56.200217 3505 tablet_replica.cc:330] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3: stopping tablet replica
I20260806 22:41:56.200413 3505 raft_consensus.cc:2229] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:56.200774 3505 raft_consensus.cc:2258] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:56.202066 3505 ts_tablet_manager.cc:1905] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:56.209026 3227 tablet_service.cc:1467] Processing CreateTablet for tablet a48de85fdbc6452d917190fc84855821 (DEFAULT_TABLE table=test-table [id=c11f76d9c03b4924b8e1e5a9b5a7882c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:56.209612 3227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48de85fdbc6452d917190fc84855821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:56.209623 3023 tablet_service.cc:1467] Processing CreateTablet for tablet a48de85fdbc6452d917190fc84855821 (DEFAULT_TABLE table=test-table [id=c11f76d9c03b4924b8e1e5a9b5a7882c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:56.210089 3023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48de85fdbc6452d917190fc84855821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:56.213683 3506 ts_tablet_manager.cc:1918] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:56.213888 3506 log.cc:1196] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/19222552e9af4291849ba8ecd84d61ea
I20260806 22:41:56.214547 3506 ts_tablet_manager.cc:1939] T 19222552e9af4291849ba8ecd84d61ea P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting consensus metadata
I20260806 22:41:56.215037 3500 tablet_replica.cc:330] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce: stopping tablet replica
I20260806 22:41:56.215265 3500 raft_consensus.cc:2229] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:56.215579 3500 raft_consensus.cc:2258] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:56.216116 2566 catalog_manager.cc:4841] TS 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933): tablet 19222552e9af4291849ba8ecd84d61ea (table test-table [id=936ddf08fd8b4ebeb71a91fe986f8f73]) successfully deleted
I20260806 22:41:56.217090 3371 tablet_bootstrap.cc:492] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce: Bootstrap starting.
I20260806 22:41:56.217034 2815 tablet_service.cc:1467] Processing CreateTablet for tablet a48de85fdbc6452d917190fc84855821 (DEFAULT_TABLE table=test-table [id=c11f76d9c03b4924b8e1e5a9b5a7882c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:41:56.217486 2815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48de85fdbc6452d917190fc84855821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:41:56.218916 3371 tablet_bootstrap.cc:654] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:56.222321 3500 ts_tablet_manager.cc:1905] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:56.222692 3372 tablet_bootstrap.cc:492] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37: Bootstrap starting.
I20260806 22:41:56.224279 3372 tablet_bootstrap.cc:654] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:56.226780 3372 tablet_bootstrap.cc:492] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37: No bootstrap required, opened a new log
I20260806 22:41:56.227034 3372 ts_tablet_manager.cc:1397] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 22:41:56.227952 3372 raft_consensus.cc:348] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.228286 3372 raft_consensus.cc:374] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:56.228427 3372 raft_consensus.cc:729] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9e43e3397a4b5e915942b7dcad3d37, State: Initialized, Role: FOLLOWER
I20260806 22:41:56.228765 3372 consensus_queue.cc:260] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.228991 3500 ts_tablet_manager.cc:1918] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:56.229151 3500 log.cc:1196] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/19222552e9af4291849ba8ecd84d61ea
I20260806 22:41:56.229749 3500 ts_tablet_manager.cc:1939] T 19222552e9af4291849ba8ecd84d61ea P a523f5ba05294594b1fdb3d7c74e58ce: Deleting consensus metadata
I20260806 22:41:56.230921 2565 catalog_manager.cc:4841] TS a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131:39757): tablet 19222552e9af4291849ba8ecd84d61ea (table test-table [id=936ddf08fd8b4ebeb71a91fe986f8f73]) successfully deleted
I20260806 22:41:56.231853 3374 tablet_bootstrap.cc:492] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3: Bootstrap starting.
I20260806 22:41:56.232208 3371 tablet_bootstrap.cc:492] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce: No bootstrap required, opened a new log
I20260806 22:41:56.232358 3371 ts_tablet_manager.cc:1397] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20260806 22:41:56.232903 3505 ts_tablet_manager.cc:1918] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:41:56.233040 3505 log.cc:1196] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/19222552e9af4291849ba8ecd84d61ea
I20260806 22:41:56.233140 3371 raft_consensus.cc:348] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.233381 3371 raft_consensus.cc:374] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:56.233457 3371 raft_consensus.cc:729] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a523f5ba05294594b1fdb3d7c74e58ce, State: Initialized, Role: FOLLOWER
I20260806 22:41:56.233503 3505 ts_tablet_manager.cc:1939] T 19222552e9af4291849ba8ecd84d61ea P aec4647a38b544e8a20c0dd669addce3: Deleting consensus metadata
I20260806 22:41:56.233664 3374 tablet_bootstrap.cc:654] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3: Neither blocks nor log segments found. Creating new log.
I20260806 22:41:56.233700 3371 consensus_queue.cc:260] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.234196 3371 ts_tablet_manager.cc:1428] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 22:41:56.235116 2566 catalog_manager.cc:4841] TS aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119): tablet 19222552e9af4291849ba8ecd84d61ea (table test-table [id=936ddf08fd8b4ebeb71a91fe986f8f73]) successfully deleted
W20260806 22:41:56.235595 2566 catalog_manager.cc:4808] TS aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119): delete failed for tablet 19222552e9af4291849ba8ecd84d61ea because the tablet was not found. No further retry: Not found: Tablet not found: 19222552e9af4291849ba8ecd84d61ea
I20260806 22:41:56.237143 3372 ts_tablet_manager.cc:1428] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20260806 22:41:56.245167 3374 tablet_bootstrap.cc:492] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3: No bootstrap required, opened a new log
I20260806 22:41:56.245473 3374 ts_tablet_manager.cc:1397] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20260806 22:41:56.246196 3374 raft_consensus.cc:348] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.246440 3374 raft_consensus.cc:374] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:41:56.246521 3374 raft_consensus.cc:729] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec4647a38b544e8a20c0dd669addce3, State: Initialized, Role: FOLLOWER
I20260806 22:41:56.246783 3374 consensus_queue.cc:260] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.248567 3374 ts_tablet_manager.cc:1428] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260806 22:41:56.395036 3381 raft_consensus.cc:482] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:41:56.395280 3381 raft_consensus.cc:504] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.396018 3381 leader_election.cc:288] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119), 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933)
I20260806 22:41:56.396607 2850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48de85fdbc6452d917190fc84855821" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec4647a38b544e8a20c0dd669addce3" is_pre_election: true
I20260806 22:41:56.396652 3050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48de85fdbc6452d917190fc84855821" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" is_pre_election: true
I20260806 22:41:56.396865 2850 raft_consensus.cc:2454] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 0.
I20260806 22:41:56.396991 3050 raft_consensus.cc:2454] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 0.
I20260806 22:41:56.397419 3157 leader_election.cc:302] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a523f5ba05294594b1fdb3d7c74e58ce, aec4647a38b544e8a20c0dd669addce3; no voters:
I20260806 22:41:56.397729 3381 raft_consensus.cc:2781] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:41:56.397868 3381 raft_consensus.cc:482] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:41:56.398005 3381 raft_consensus.cc:3037] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:56.399960 3381 raft_consensus.cc:504] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.400525 3381 leader_election.cc:288] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 election: Requested vote from peers aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119), 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933)
I20260806 22:41:56.400935 2850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48de85fdbc6452d917190fc84855821" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec4647a38b544e8a20c0dd669addce3"
I20260806 22:41:56.401032 3050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48de85fdbc6452d917190fc84855821" candidate_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a9e43e3397a4b5e915942b7dcad3d37"
I20260806 22:41:56.401165 2850 raft_consensus.cc:3037] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:56.401266 3050 raft_consensus.cc:3037] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:41:56.403303 2850 raft_consensus.cc:2454] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 1.
I20260806 22:41:56.403313 3050 raft_consensus.cc:2454] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a523f5ba05294594b1fdb3d7c74e58ce in term 1.
I20260806 22:41:56.403743 3157 leader_election.cc:302] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a523f5ba05294594b1fdb3d7c74e58ce, aec4647a38b544e8a20c0dd669addce3; no voters:
I20260806 22:41:56.404017 3381 raft_consensus.cc:2781] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:41:56.404412 3381 raft_consensus.cc:686] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 LEADER]: Becoming Leader. State: Replica: a523f5ba05294594b1fdb3d7c74e58ce, State: Running, Role: LEADER
I20260806 22:41:56.404671 3381 consensus_queue.cc:237] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } }
I20260806 22:41:56.407230 2586 catalog_manager.cc:5495] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce reported cstate change: term changed from 0 to 1, leader changed from <none> to a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131). New cstate: current_term: 1 leader_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a523f5ba05294594b1fdb3d7c74e58ce" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 39757 } health_report { overall_health: HEALTHY } } }
I20260806 22:41:56.433306 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:56.427733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.23.100.131:39757)
0806 22:41:56.428272 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:56.428276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:56.428311 (+ 35us) server_negotiation.cc:407] Connection header received
0806 22:41:56.428411 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:56.428415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:56.428481 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:56.428564 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:56.428955 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.429752 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.431378 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.431792 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.431889 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:56.432271 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:56.432388 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:56.432454 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:56.432854 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 22:41:56.432918 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
I20260806 22:41:56.446269 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:56.441388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:56961 (local address 127.23.100.131:39757)
0806 22:41:56.441612 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:56.441622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:56.442029 (+ 407us) server_negotiation.cc:407] Connection header received
0806 22:41:56.442160 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:56.442163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:56.442222 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:56.442307 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:41:56.442801 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.443492 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.444756 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.445168 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.445262 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:56.445698 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:41:56.445794 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:41:56.445864 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:56.445987 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 22:41:56.446049 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260806 22:41:56.446434 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:56.441301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:39757 (local address 127.23.100.131:56961)
0806 22:41:56.441925 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0806 22:41:56.441940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 22:41:56.442062 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:41:56.442334 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:41:56.442337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:41:56.442346 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 22:41:56.442642 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:56.442649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.443597 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.443601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:56.444626 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:41:56.444638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.445321 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.445325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:41:56.445444 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:56.445481 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 22:41:56.445881 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 22:41:56.445890 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 22:41:56.445958 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 22:41:56.446059 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":18,"thread_start_us":385,"threads_started":1}
I20260806 22:41:56.454479 2850 raft_consensus.cc:1261] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Refusing update from remote peer a523f5ba05294594b1fdb3d7c74e58ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:56.454481 3050 raft_consensus.cc:1261] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Refusing update from remote peer a523f5ba05294594b1fdb3d7c74e58ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:41:56.455031 3517 consensus_queue.cc:1035] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec4647a38b544e8a20c0dd669addce3" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:56.455364 3519 consensus_queue.cc:1035] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a9e43e3397a4b5e915942b7dcad3d37" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:41:56.501082 2586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:56.501351 2586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:41:56.504271 2586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dee621521a974961bbd1844e8249b0e9: Sending DeleteTablet for 3 replicas of tablet a48de85fdbc6452d917190fc84855821
I20260806 22:41:56.505069 2815 tablet_service.cc:1514] Processing DeleteTablet for tablet a48de85fdbc6452d917190fc84855821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:56 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:47984
I20260806 22:41:56.505194 3227 tablet_service.cc:1514] Processing DeleteTablet for tablet a48de85fdbc6452d917190fc84855821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:56 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:48538
I20260806 22:41:56.505173 3023 tablet_service.cc:1514] Processing DeleteTablet for tablet a48de85fdbc6452d917190fc84855821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:41:56 UTC) from {username='oryx', principal='oryx/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:45864
I20260806 22:41:56.505360 3505 tablet_replica.cc:330] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3: stopping tablet replica
I20260806 22:41:56.505465 3506 tablet_replica.cc:330] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37: stopping tablet replica
I20260806 22:41:56.505622 3505 raft_consensus.cc:2229] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:56.505697 3506 raft_consensus.cc:2229] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:41:56.505846 3505 raft_consensus.cc:2258] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:56.505950 3506 raft_consensus.cc:2258] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:56.506114 3500 tablet_replica.cc:330] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce: stopping tablet replica
I20260806 22:41:56.506316 3500 raft_consensus.cc:2229] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:41:56.506704 3500 raft_consensus.cc:2258] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:41:56.507077 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 2708
I20260806 22:41:56.507210 3506 ts_tablet_manager.cc:1905] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:56.507972 3500 ts_tablet_manager.cc:1905] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:56.513096 3500 ts_tablet_manager.cc:1918] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:56.513283 3500 log.cc:1196] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/a48de85fdbc6452d917190fc84855821
I20260806 22:41:56.514343 3505 ts_tablet_manager.cc:1905] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:41:56.514911 3500 ts_tablet_manager.cc:1939] T a48de85fdbc6452d917190fc84855821 P a523f5ba05294594b1fdb3d7c74e58ce: Deleting consensus metadata
I20260806 22:41:56.515570 3506 ts_tablet_manager.cc:1918] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:56.515740 3506 log.cc:1196] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/a48de85fdbc6452d917190fc84855821
I20260806 22:41:56.516186 2565 catalog_manager.cc:4841] TS a523f5ba05294594b1fdb3d7c74e58ce (127.23.100.131:39757): tablet a48de85fdbc6452d917190fc84855821 (table test-table [id=c11f76d9c03b4924b8e1e5a9b5a7882c]) successfully deleted
I20260806 22:41:56.516270 3506 ts_tablet_manager.cc:1939] T a48de85fdbc6452d917190fc84855821 P 7a9e43e3397a4b5e915942b7dcad3d37: Deleting consensus metadata
I20260806 22:41:56.517572 2566 catalog_manager.cc:4841] TS 7a9e43e3397a4b5e915942b7dcad3d37 (127.23.100.130:39933): tablet a48de85fdbc6452d917190fc84855821 (table test-table [id=c11f76d9c03b4924b8e1e5a9b5a7882c]) successfully deleted
I20260806 22:41:56.520709 3505 ts_tablet_manager.cc:1918] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 22:41:56.520885 3505 log.cc:1196] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/a48de85fdbc6452d917190fc84855821
I20260806 22:41:56.521378 3505 ts_tablet_manager.cc:1939] T a48de85fdbc6452d917190fc84855821 P aec4647a38b544e8a20c0dd669addce3: Deleting consensus metadata
I20260806 22:41:56.522619 2566 catalog_manager.cc:4841] TS aec4647a38b544e8a20c0dd669addce3 (127.23.100.129:46119): tablet a48de85fdbc6452d917190fc84855821 (table test-table [id=c11f76d9c03b4924b8e1e5a9b5a7882c]) successfully deleted
Aug 06 22:41:56 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.129@KRBTEST.COM
I20260806 22:41:56.531062 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:56.508447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.23.100.129:46119)
0806 22:41:56.508628 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:56.508634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:56.514342 (+ 5708us) server_negotiation.cc:407] Connection header received
0806 22:41:56.514579 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:56.514585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:56.514676 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:56.514781 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:56.516968 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.518080 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.519905 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.520302 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.520406 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:56.524869 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:56.524887 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:56.524904 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:56.524935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:56.528120 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:56.528763 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:56.528770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:56.528772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:56.528895 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:56.529267 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:56.529271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:56.529273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:56.529780 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:56.530046 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:56.530551 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 22:41:56.530733 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 22:41:56.531798 2908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:56.660247 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 2708
I20260806 22:41:56.691811 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 2926
Aug 06 22:41:56 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.130@KRBTEST.COM
I20260806 22:41:56.707135 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:56.692705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.23.100.130:39933)
0806 22:41:56.693206 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:56.693211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:56.693228 (+ 17us) server_negotiation.cc:407] Connection header received
0806 22:41:56.693284 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:56.693288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:56.693347 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:56.693469 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:56.694802 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.695632 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.696816 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.697229 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.697341 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:56.701454 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:56.701470 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:56.701486 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:56.701577 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:56.704593 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:56.705246 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:56.705254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:56.705256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:56.705386 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:56.705737 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:56.705740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:56.705742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:56.706115 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:56.706359 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:56.706564 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 22:41:56.706735 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":251,"threads_started":1}
I20260806 22:41:56.707826 3117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:56.807091 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 2926
I20260806 22:41:56.834219 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 3129
Aug 06 22:41:56 dist-test-slave-zr1t krb5kdc[2523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.100.131@KRBTEST.COM
I20260806 22:41:56.850427 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:41:56.835023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.23.100.131:39757)
0806 22:41:56.835241 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 22:41:56.835249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:41:56.835326 (+ 77us) server_negotiation.cc:407] Connection header received
0806 22:41:56.835549 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:41:56.835559 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:41:56.835623 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:41:56.835732 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:41:56.837089 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.837877 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.839180 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:41:56.839633 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:41:56.839775 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:41:56.844352 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:41:56.844374 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:41:56.844391 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:41:56.844434 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:41:56.847499 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:56.848218 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:56.848227 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:56.848229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:56.848377 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:41:56.848766 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:41:56.848771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:41:56.848774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:41:56.849139 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:41:56.849511 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0806 22:41:56.849831 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 22:41:56.850063 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 22:41:56.851203 3328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:57.006417 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 3129
I20260806 22:41:57.038908 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 2542
I20260806 22:41:57.040061 2655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:41:57.161890 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 2542
2026-08-06T22:41:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6154 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:41:57 dist-test-slave-zr1t krb5kdc[3548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:41:57 dist-test-slave-zr1t krb5kdc[3548](info): set up 2 sockets
Aug 06 22:41:57 dist-test-slave-zr1t krb5kdc[3548](info): commencing operation
krb5kdc: starting...
W20260806 22:41:58.508778 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:41:58 dist-test-slave-zr1t krb5kdc[3548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:41:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:41:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:41:58.806088 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:36861
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:36861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:41:59.150027 3572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:41:59.150383 3572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:41:59.150542 3572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:41:59.157974 3572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:41:59.158139 3572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:41:59.158262 3572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:41:59.158401 3572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:41:59.158500 3572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:41:59.169019 3572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:36861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:36861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:41:59.170740 3572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:41:59.172145 3572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:41:59.184834 3583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:59.184894 3580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:41:59.184916 3581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:41:59.185853 3572 server_base.cc:1034] running on GCE node
I20260806 22:41:59.186700 3572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:41:59.188220 3572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:41:59.189481 3572 hybrid_clock.cc:648] HybridClock initialized: now 1786056119189436 us; error 55 us; skew 500 ppm
I20260806 22:41:59.189927 3572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T22:41:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2100 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:41:59 dist-test-slave-zr1t krb5kdc[3596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:41:59 dist-test-slave-zr1t krb5kdc[3596](info): set up 2 sockets
Aug 06 22:41:59 dist-test-slave-zr1t krb5kdc[3596](info): commencing operation
krb5kdc: starting...
W20260806 22:42:00.586337 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:42:00 dist-test-slave-zr1t krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:42:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:42:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:42:00.891175 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33557
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:42:01.239238 3616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:01.239549 3616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:01.239722 3616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:01.247007 3616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:42:01.247138 3616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:01.247248 3616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:01.247350 3616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:42:01.247437 3616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:42:01.257372 3616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:01.258796 3616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:01.260098 3616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:01.272039 3625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:01.272079 3631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:01.272130 3623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:01.273223 3616 server_base.cc:1034] running on GCE node
I20260806 22:42:01.274057 3616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:01.275524 3616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:01.276762 3616 hybrid_clock.cc:648] HybridClock initialized: now 1786056121276723 us; error 51 us; skew 500 ppm
I20260806 22:42:01.277122 3616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:01 dist-test-slave-zr1t krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056121, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:01.280335 3616 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:42:01.281940 3616 webserver.cc:460] Webserver started at http://127.23.100.190:41883/ using document root <none> and password file <none>
I20260806 22:42:01.282648 3616 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:01.282797 3616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:01.283118 3616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:01.286194 3616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "a061e50892684eef8975caba1a8f697d"
format_stamp: "Formatted at 2026-08-06 22:42:01 on dist-test-slave-zr1t"
server_key: "6b308e59d91e1d0fdeb28a01d4cec725"
server_key_iv: "27e13800aa65d6006dd18d1bf701ac66"
server_key_version: "encryptionkey@0"
I20260806 22:42:01.286949 3616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "a061e50892684eef8975caba1a8f697d"
format_stamp: "Formatted at 2026-08-06 22:42:01 on dist-test-slave-zr1t"
server_key: "6b308e59d91e1d0fdeb28a01d4cec725"
server_key_iv: "27e13800aa65d6006dd18d1bf701ac66"
server_key_version: "encryptionkey@0"
I20260806 22:42:01.292062 3616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:42:01.295491 3644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:01.296766 3616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:42:01.296988 3616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "a061e50892684eef8975caba1a8f697d"
format_stamp: "Formatted at 2026-08-06 22:42:01 on dist-test-slave-zr1t"
server_key: "6b308e59d91e1d0fdeb28a01d4cec725"
server_key_iv: "27e13800aa65d6006dd18d1bf701ac66"
server_key_version: "encryptionkey@0"
I20260806 22:42:01.297215 3616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:01.322923 3616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:01.327266 3616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:01.327589 3616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:01.347210 3616 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:33557
I20260806 22:42:01.347249 3734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:33557 every 8 connection(s)
I20260806 22:42:01.348660 3616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:42:01.352098 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 3616
I20260806 22:42:01.352331 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:42:01.352833 3736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:42:01.353209 23954 external_mini_cluster.cc:1428] Setting key 411aa473f3343725f498a02bfee4ed0f
I20260806 22:42:01.368213 3736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d: Bootstrap starting.
Aug 06 22:42:01 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:42:01.372463 3736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d: Neither blocks nor log segments found. Creating new log.
I20260806 22:42:01.373979 3736 log.cc:824] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d: Log is configured to *not* fsync() on all Append() calls
I20260806 22:42:01.378891 3736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d: No bootstrap required, opened a new log
I20260806 22:42:01.379594 3740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:01.355304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.23.100.190:33557)
0806 22:42:01.356160 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:01.356171 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:01.356210 (+ 39us) server_negotiation.cc:407] Connection header received
0806 22:42:01.357897 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:01.357919 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:01.358297 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:01.358621 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:01.360545 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:01.361971 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:01.363853 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:01.364547 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:01.364689 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:01.369608 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:01.369620 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:01.369639 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:01.369700 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:01.374045 (+ 4345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:01.375154 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:01.375167 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:01.375179 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:01.375472 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:01.376100 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:01.376111 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:01.376117 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:01.376856 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:01.377153 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:01.377747 (+ 594us) server_negotiation.cc:299] Negotiation successful
0806 22:42:01.378188 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":369,"threads_started":1}
I20260806 22:42:01.388964 3736 raft_consensus.cc:348] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } }
I20260806 22:42:01.389351 3736 raft_consensus.cc:374] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:42:01.389432 3736 raft_consensus.cc:729] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a061e50892684eef8975caba1a8f697d, State: Initialized, Role: FOLLOWER
I20260806 22:42:01.390121 3736 consensus_queue.cc:260] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } }
I20260806 22:42:01.390368 3736 raft_consensus.cc:388] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:42:01.390529 3736 raft_consensus.cc:482] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:42:01.390726 3736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:42:01.392992 3736 raft_consensus.cc:504] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } }
I20260806 22:42:01.393605 3736 leader_election.cc:302] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a061e50892684eef8975caba1a8f697d; no voters:
I20260806 22:42:01.394236 3736 leader_election.cc:288] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:42:01.394387 3742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:42:01.395094 3742 raft_consensus.cc:686] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 1 LEADER]: Becoming Leader. State: Replica: a061e50892684eef8975caba1a8f697d, State: Running, Role: LEADER
I20260806 22:42:01.395648 3742 consensus_queue.cc:237] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } }
I20260806 22:42:01.396482 3736 sys_catalog.cc:564] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:42:01.398677 3746 sys_catalog.cc:455] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a061e50892684eef8975caba1a8f697d. Latest consensus state: current_term: 1 leader_uuid: "a061e50892684eef8975caba1a8f697d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } } }
I20260806 22:42:01.399165 3746 sys_catalog.cc:458] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [sys.catalog]: This master's current role is: LEADER
I20260806 22:42:01.400408 3745 sys_catalog.cc:455] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a061e50892684eef8975caba1a8f697d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } } }
I20260806 22:42:01.401084 3745 sys_catalog.cc:458] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [sys.catalog]: This master's current role is: LEADER
I20260806 22:42:01.401988 3757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:42:01.407327 3757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:42:01.418553 3757 catalog_manager.cc:1347] Generated new cluster ID: a9b194bc98234cb489917d9c775cf297
I20260806 22:42:01.418713 3757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:42:01.442754 3757 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:42:01.444186 3757 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:42:01.451480 3757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d: Generated new TSK 0
I20260806 22:42:01.452405 3757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:42:01.537258 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33557
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:42:01.892251 3781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:01.892540 3781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:01.892637 3781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:01.900363 3781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:01.900486 3781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:01.900660 3781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:42:01.910825 3781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:01.912196 3781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:01.913568 3781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:01.927778 3792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:01.927713 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:01.927755 3791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:01.928872 3781 server_base.cc:1034] running on GCE node
I20260806 22:42:01.929577 3781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:01.930565 3781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:01.931830 3781 hybrid_clock.cc:648] HybridClock initialized: now 1786056121931779 us; error 40 us; skew 500 ppm
I20260806 22:42:01.932204 3781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:01 dist-test-slave-zr1t krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056121, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:01.935674 3781 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:42:01.937316 3781 webserver.cc:460] Webserver started at http://127.23.100.129:35785/ using document root <none> and password file <none>
I20260806 22:42:01.938091 3781 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:01.938242 3781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:01.938576 3781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:01.941596 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "f9970751ccc4410cb400f0f7e5da02c1"
format_stamp: "Formatted at 2026-08-06 22:42:01 on dist-test-slave-zr1t"
server_key: "67207ff8b66170ba36418483c6e893cc"
server_key_iv: "acd047860c74006e99a9449eec674313"
server_key_version: "encryptionkey@0"
I20260806 22:42:01.942441 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "f9970751ccc4410cb400f0f7e5da02c1"
format_stamp: "Formatted at 2026-08-06 22:42:01 on dist-test-slave-zr1t"
server_key: "67207ff8b66170ba36418483c6e893cc"
server_key_iv: "acd047860c74006e99a9449eec674313"
server_key_version: "encryptionkey@0"
I20260806 22:42:01.947795 3781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:42:01.951392 3809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:01.952759 3781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 22:42:01.952943 3781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "f9970751ccc4410cb400f0f7e5da02c1"
format_stamp: "Formatted at 2026-08-06 22:42:01 on dist-test-slave-zr1t"
server_key: "67207ff8b66170ba36418483c6e893cc"
server_key_iv: "acd047860c74006e99a9449eec674313"
server_key_version: "encryptionkey@0"
I20260806 22:42:01.953155 3781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:01.969743 3781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:01.974051 3781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:01.974346 3781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:01.975355 3781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:01.976728 3781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:01.976835 3781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:01.976984 3781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:01.977062 3781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:02.011345 3781 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:34519
I20260806 22:42:02.011384 3991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:34519 every 8 connection(s)
I20260806 22:42:02.012662 3781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:42:02.019001 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 3781
I20260806 22:42:02.019222 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:42:02.019869 23954 external_mini_cluster.cc:1428] Setting key 4d0a55d29c4b5a901c6baea9ecc2b9e6
Aug 06 22:42:02 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056121, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:42:02.034811 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:02.015249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:44683 (local address 127.23.100.190:33557)
0806 22:42:02.015744 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:02.015748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:02.016622 (+ 874us) server_negotiation.cc:407] Connection header received
0806 22:42:02.018707 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:02.018713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:02.018806 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:02.018927 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:02.020652 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:02.021686 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:02.023326 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:02.024094 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:02.024258 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:02.027923 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:02.027932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:02.027940 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:02.027989 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:02.031712 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:02.032458 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:02.032467 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:02.032472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:02.032633 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:02.033049 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:02.033056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:02.033062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:02.033459 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:02.033650 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:02.034361 (+ 711us) server_negotiation.cc:299] Negotiation successful
0806 22:42:02.034538 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":268,"threads_started":1}
I20260806 22:42:02.036376 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:02.015698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:44683)
0806 22:42:02.016456 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:02.016501 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:02.018441 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:02.019204 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:02.019216 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:02.019779 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:02.020458 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:02.020485 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:02.021862 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:02.021870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:02.023155 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:02.023168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:02.024310 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:02.024315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:02.024516 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:02.025208 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:02.025231 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:02.027692 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:02.031868 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:02.031876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:02.031890 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:02.032334 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:02.032744 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:02.032747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:02.032750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:02.032928 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:02.033624 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:02.033630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:02.033959 (+ 329us) client_negotiation.cc:769] Sending connection context
0806 22:42:02.034199 (+ 240us) client_negotiation.cc:240] Negotiation successful
0806 22:42:02.034812 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":291,"threads_started":1}
I20260806 22:42:02.038010 3992 heartbeater.cc:347] Connected to a master server at 127.23.100.190:33557
I20260806 22:42:02.038364 3992 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:02.039100 3992 heartbeater.cc:511] Master 127.23.100.190:33557 requested a full tablet report, sending...
I20260806 22:42:02.041342 3677 ts_manager.cc:194] Registered new tserver with Master: f9970751ccc4410cb400f0f7e5da02c1 (127.23.100.129:34519)
I20260806 22:42:02.042919 3677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:44683
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:42:02.084707 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33557
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:42:02.428164 4002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:02.428488 4002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:02.428664 4002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:02.436010 4002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:02.436167 4002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:02.436353 4002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:42:02.446449 4002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:02.447788 4002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:02.449278 4002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:02.462898 4009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:02.462852 4010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:02.462818 4013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:02.464097 4002 server_base.cc:1034] running on GCE node
I20260806 22:42:02.464743 4002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:02.465767 4002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:02.466977 4002 hybrid_clock.cc:648] HybridClock initialized: now 1786056122466926 us; error 35 us; skew 500 ppm
I20260806 22:42:02.467299 4002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:02 dist-test-slave-zr1t krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056122, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:02.470713 4002 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:42:02.472287 4002 webserver.cc:460] Webserver started at http://127.23.100.130:37993/ using document root <none> and password file <none>
I20260806 22:42:02.473097 4002 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:02.473235 4002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:02.473637 4002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:02.476603 4002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "23bacee2e94a41929d08be1f48f03681"
format_stamp: "Formatted at 2026-08-06 22:42:02 on dist-test-slave-zr1t"
server_key: "da1c5a7fadf4f6f678ffe22c56be14c1"
server_key_iv: "1b0ebb05b6c868868ed831fe9bbc293e"
server_key_version: "encryptionkey@0"
I20260806 22:42:02.477427 4002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "23bacee2e94a41929d08be1f48f03681"
format_stamp: "Formatted at 2026-08-06 22:42:02 on dist-test-slave-zr1t"
server_key: "da1c5a7fadf4f6f678ffe22c56be14c1"
server_key_iv: "1b0ebb05b6c868868ed831fe9bbc293e"
server_key_version: "encryptionkey@0"
I20260806 22:42:02.482861 4002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 22:42:02.486457 4026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:02.487833 4002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:42:02.488034 4002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "23bacee2e94a41929d08be1f48f03681"
format_stamp: "Formatted at 2026-08-06 22:42:02 on dist-test-slave-zr1t"
server_key: "da1c5a7fadf4f6f678ffe22c56be14c1"
server_key_iv: "1b0ebb05b6c868868ed831fe9bbc293e"
server_key_version: "encryptionkey@0"
I20260806 22:42:02.488265 4002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:02.505712 4002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:02.510179 4002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:02.510504 4002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:02.511487 4002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:02.512905 4002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:02.513027 4002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:02.513191 4002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:02.513265 4002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:02.548732 4002 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:40839
I20260806 22:42:02.548820 4214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:40839 every 8 connection(s)
I20260806 22:42:02.550081 4002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:42:02.554768 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 4002
I20260806 22:42:02.554956 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:42:02.555588 23954 external_mini_cluster.cc:1428] Setting key f036705587dedcdc52d5c8067c943eeb
Aug 06 22:42:02 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056122, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:42:02.574517 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:02.552769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:52335 (local address 127.23.100.190:33557)
0806 22:42:02.553237 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:02.553242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:02.554274 (+ 1032us) server_negotiation.cc:407] Connection header received
0806 22:42:02.556974 (+ 2700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:02.556980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:02.557058 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:02.557177 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:02.559339 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:02.560052 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:02.562597 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:02.563031 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:02.563127 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:02.567718 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:02.567724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:02.567727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:02.567763 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:02.570886 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:02.571649 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:02.571658 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:02.571664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:02.571831 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:02.572315 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:02.572323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:02.572329 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:02.572725 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:02.572879 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:02.573731 (+ 852us) server_negotiation.cc:299] Negotiation successful
0806 22:42:02.574032 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":236,"threads_started":1}
I20260806 22:42:02.575182 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:02.553322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:52335)
0806 22:42:02.554102 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:02.554143 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:02.556637 (+ 2494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:02.557460 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:02.557475 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:02.558278 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:02.559132 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:02.559161 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:02.560225 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:02.560234 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:02.562441 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:02.562461 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:02.563264 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:02.563273 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:02.563612 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:02.564390 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:02.564426 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:02.567463 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:02.571052 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:02.571059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:02.571072 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:02.571505 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:02.571977 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:02.571981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:02.571983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:02.572185 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:02.572849 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:02.572855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:02.573295 (+ 440us) client_negotiation.cc:769] Sending connection context
0806 22:42:02.573626 (+ 331us) client_negotiation.cc:240] Negotiation successful
0806 22:42:02.574053 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":339,"threads_started":1}
I20260806 22:42:02.576797 4217 heartbeater.cc:347] Connected to a master server at 127.23.100.190:33557
I20260806 22:42:02.577140 4217 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:02.577963 4217 heartbeater.cc:511] Master 127.23.100.190:33557 requested a full tablet report, sending...
I20260806 22:42:02.579463 3677 ts_manager.cc:194] Registered new tserver with Master: 23bacee2e94a41929d08be1f48f03681 (127.23.100.130:40839)
I20260806 22:42:02.580297 3677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:52335
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:42:02.620791 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33557
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:42:02.967533 4230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:02.967818 4230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:02.967981 4230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:02.975586 4230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:02.975708 4230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:02.975917 4230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:42:02.986207 4230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:02.987537 4230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:02.988847 4230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:03.002571 4238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:03.002607 4244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:03.002535 4241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:03.003821 4230 server_base.cc:1034] running on GCE node
I20260806 22:42:03.004568 4230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:03.005663 4230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:03.006997 4230 hybrid_clock.cc:648] HybridClock initialized: now 1786056123006917 us; error 56 us; skew 500 ppm
I20260806 22:42:03.007396 4230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:03 dist-test-slave-zr1t krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056123, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:03.010959 4230 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:42:03.012691 4230 webserver.cc:460] Webserver started at http://127.23.100.131:43523/ using document root <none> and password file <none>
I20260806 22:42:03.013406 4230 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:03.013578 4230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:03.013906 4230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:03.016909 4230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "e0721e4ecc70479190c0334b53daa937"
format_stamp: "Formatted at 2026-08-06 22:42:03 on dist-test-slave-zr1t"
server_key: "ffa4e8c882b187396ca06d7175818893"
server_key_iv: "13a2b40581cddb92b2d8fb10193569e1"
server_key_version: "encryptionkey@0"
I20260806 22:42:03.017792 4230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "e0721e4ecc70479190c0334b53daa937"
format_stamp: "Formatted at 2026-08-06 22:42:03 on dist-test-slave-zr1t"
server_key: "ffa4e8c882b187396ca06d7175818893"
server_key_iv: "13a2b40581cddb92b2d8fb10193569e1"
server_key_version: "encryptionkey@0"
I20260806 22:42:03.023046 4230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:42:03.026496 4253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:03.027769 4230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:42:03.027949 4230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "e0721e4ecc70479190c0334b53daa937"
format_stamp: "Formatted at 2026-08-06 22:42:03 on dist-test-slave-zr1t"
server_key: "ffa4e8c882b187396ca06d7175818893"
server_key_iv: "13a2b40581cddb92b2d8fb10193569e1"
server_key_version: "encryptionkey@0"
I20260806 22:42:03.028160 4230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:03.045830 3992 heartbeater.cc:503] Master 127.23.100.190:33557 was elected leader, sending a full tablet report...
I20260806 22:42:03.049409 4230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:03.054042 4230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:03.054296 4230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:03.055251 4230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:03.056851 4230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:03.056931 4230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:03.057024 4230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:03.057070 4230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:03.092130 4230 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:45347
I20260806 22:42:03.092149 4423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:45347 every 8 connection(s)
I20260806 22:42:03.093499 4230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:42:03.099864 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 4230
I20260806 22:42:03.100081 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:42:03.100597 23954 external_mini_cluster.cc:1428] Setting key d58ec2e2a89bad13468a475b5faba2b9
Aug 06 22:42:03 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056123, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:42:03.115062 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:03.096120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:46851 (local address 127.23.100.190:33557)
0806 22:42:03.096637 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:03.096641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:03.097805 (+ 1164us) server_negotiation.cc:407] Connection header received
0806 22:42:03.099685 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:03.099691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:03.099783 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:03.099903 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:03.101945 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:03.103122 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:03.104891 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:03.105352 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:03.105462 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:03.109444 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:03.109450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:03.109453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:03.109487 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:03.112083 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:03.112776 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:03.112780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:03.112782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:03.112927 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:03.113299 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:03.113302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:03.113304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:03.113642 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:03.113769 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:03.114442 (+ 673us) server_negotiation.cc:299] Negotiation successful
0806 22:42:03.114656 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":253,"threads_started":1}
I20260806 22:42:03.116158 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:03.096775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:46851)
0806 22:42:03.097625 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:03.097675 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:03.099390 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:03.100143 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:03.100173 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:03.100926 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:03.101738 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:03.101765 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:03.103281 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:03.103291 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:03.104738 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:03.104752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:03.105634 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:03.105641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:03.105853 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:03.106570 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:03.106597 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:03.109223 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:03.112217 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:03.112224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:03.112237 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:03.112662 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:03.113022 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:03.113025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:03.113027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:03.113200 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:03.113759 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:03.113765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:03.114077 (+ 312us) client_negotiation.cc:769] Sending connection context
0806 22:42:03.114299 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 22:42:03.114803 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":275,"threads_started":1}
I20260806 22:42:03.117439 4424 heartbeater.cc:347] Connected to a master server at 127.23.100.190:33557
I20260806 22:42:03.117760 4424 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:03.118438 4424 heartbeater.cc:511] Master 127.23.100.190:33557 requested a full tablet report, sending...
I20260806 22:42:03.119755 3677 ts_manager.cc:194] Registered new tserver with Master: e0721e4ecc70479190c0334b53daa937 (127.23.100.131:45347)
I20260806 22:42:03.120518 3677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:46851
I20260806 22:42:03.128690 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab.
Entry for principal oryx/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab.
WARNING: no policy specified for oryx/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab.
Entry for principal oryx/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab.
WARNING: no policy specified for oryx/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab.
Entry for principal oryx/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab.
I20260806 22:42:03.312422 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 3781
Aug 06 22:42:03 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:42:03.333205 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:03.313911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.23.100.129:34519)
0806 22:42:03.318038 (+ 4127us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:03.318046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:03.318074 (+ 28us) server_negotiation.cc:407] Connection header received
0806 22:42:03.318136 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:03.318142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:03.318354 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:03.318553 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:03.320053 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:03.321102 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:03.322509 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:03.323078 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:03.323223 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:03.327620 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:03.327633 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:03.327648 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:03.327688 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:03.330500 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:03.331185 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:03.331192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:03.331194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:03.331370 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:03.331793 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:03.331803 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:03.331805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:03.332150 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:03.332420 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:03.332665 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 22:42:03.332919 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3968,"thread_start_us":3813,"threads_started":1}
I20260806 22:42:03.334617 3987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:03.417434 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 3781
I20260806 22:42:03.441092 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 4002
Aug 06 22:42:03 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:42:03.457957 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:03.442143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.23.100.130:40839)
0806 22:42:03.442667 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:03.442674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:03.442700 (+ 26us) server_negotiation.cc:407] Connection header received
0806 22:42:03.442761 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:03.442768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:03.442996 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:03.443127 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:03.444318 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:03.445312 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:03.446648 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:03.447613 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:03.447760 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:03.452238 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:03.452251 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:03.452266 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:03.452303 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:03.455254 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:03.455941 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:03.455948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:03.455950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:03.456081 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:03.456385 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:03.456388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:03.456390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:03.456740 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:03.457005 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:03.457267 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 22:42:03.457482 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":275,"threads_started":1}
I20260806 22:42:03.459285 4208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:03.543915 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 4002
I20260806 22:42:03.567955 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 4230
Aug 06 22:42:03 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:42:03.585183 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:03.569050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.23.100.131:45347)
0806 22:42:03.569650 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:03.569658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:03.569690 (+ 32us) server_negotiation.cc:407] Connection header received
0806 22:42:03.569759 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:03.569766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:03.569979 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:03.570112 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:03.571544 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:03.572550 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:03.574119 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:03.574663 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:03.574759 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:03.579409 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:03.579423 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:03.579437 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:03.579476 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:03.582552 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:03.583207 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:03.583214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:03.583217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:03.583338 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:03.583670 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:03.583673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:03.583675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:03.584159 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:03.584404 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:03.584656 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 22:42:03.584863 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":222,"threads_started":1}
I20260806 22:42:03.586613 4415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:03.671614 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 4230
I20260806 22:42:03.695894 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 3616
I20260806 22:42:03.697175 3727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:03.802274 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 3616
I20260806 22:42:03.829172 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33557
--webserver_interface=127.23.100.190
--webserver_port=41883
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:42:04.179469 4461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:04.179780 4461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:04.179944 4461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:04.187577 4461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:42:04.187698 4461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:04.187806 4461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:04.187894 4461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:42:04.187979 4461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:42:04.198084 4461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=41883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:04.199501 4461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:04.200825 4461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:04.213205 4470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:04.213218 4471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:04.213243 4473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:04.214063 4461 server_base.cc:1034] running on GCE node
I20260806 22:42:04.214849 4461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:04.216302 4461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:04.217619 4461 hybrid_clock.cc:648] HybridClock initialized: now 1786056124217510 us; error 57 us; skew 500 ppm
I20260806 22:42:04.218024 4461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:04 dist-test-slave-zr1t krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056124, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:04.225268 4461 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:42:04.227177 4461 webserver.cc:460] Webserver started at http://127.23.100.190:41883/ using document root <none> and password file <none>
I20260806 22:42:04.228045 4461 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:04.228210 4461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:04.234283 4461 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 22:42:04.237303 4482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:04.238806 4461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:42:04.239004 4461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "a061e50892684eef8975caba1a8f697d"
format_stamp: "Formatted at 2026-08-06 22:42:01 on dist-test-slave-zr1t"
server_key: "6b308e59d91e1d0fdeb28a01d4cec725"
server_key_iv: "27e13800aa65d6006dd18d1bf701ac66"
server_key_version: "encryptionkey@0"
I20260806 22:42:04.240036 4461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:04.265367 4461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:04.271221 4461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:04.271552 4461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:04.292227 4461 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:33557
I20260806 22:42:04.292320 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:33557 every 8 connection(s)
I20260806 22:42:04.293808 4461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:42:04.298430 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 4461
I20260806 22:42:04.299284 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:34519
--local_ip_for_outbound_sockets=127.23.100.129
--tserver_master_addrs=127.23.100.190:33557
--webserver_port=35785
--webserver_interface=127.23.100.129
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.100.129
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:42:04.304401 4554 sys_catalog.cc:263] Verifying existing consensus state
I20260806 22:42:04.309953 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d: Bootstrap starting.
I20260806 22:42:04.339056 4554 log.cc:824] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d: Log is configured to *not* fsync() on all Append() calls
I20260806 22:42:04.348723 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 22:42:04.349272 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d: Bootstrap complete.
I20260806 22:42:04.358609 4554 raft_consensus.cc:348] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } }
I20260806 22:42:04.359504 4554 raft_consensus.cc:729] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a061e50892684eef8975caba1a8f697d, State: Initialized, Role: FOLLOWER
I20260806 22:42:04.360158 4554 consensus_queue.cc:260] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } }
I20260806 22:42:04.360421 4554 raft_consensus.cc:388] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:42:04.360553 4554 raft_consensus.cc:482] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:42:04.360731 4554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 1 FOLLOWER]: Advancing to term 2
I20260806 22:42:04.364599 4554 raft_consensus.cc:504] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } }
I20260806 22:42:04.365103 4554 leader_election.cc:302] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a061e50892684eef8975caba1a8f697d; no voters:
I20260806 22:42:04.365733 4554 leader_election.cc:288] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 22:42:04.365935 4565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 22:42:04.367282 4565 raft_consensus.cc:686] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [term 2 LEADER]: Becoming Leader. State: Replica: a061e50892684eef8975caba1a8f697d, State: Running, Role: LEADER
I20260806 22:42:04.367856 4565 consensus_queue.cc:237] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } }
I20260806 22:42:04.368021 4554 sys_catalog.cc:564] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:42:04.370697 4566 sys_catalog.cc:455] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a061e50892684eef8975caba1a8f697d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } } }
I20260806 22:42:04.371155 4566 sys_catalog.cc:458] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [sys.catalog]: This master's current role is: LEADER
I20260806 22:42:04.371706 4567 sys_catalog.cc:455] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a061e50892684eef8975caba1a8f697d. Latest consensus state: current_term: 2 leader_uuid: "a061e50892684eef8975caba1a8f697d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a061e50892684eef8975caba1a8f697d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33557 } } }
I20260806 22:42:04.372087 4567 sys_catalog.cc:458] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d [sys.catalog]: This master's current role is: LEADER
I20260806 22:42:04.379592 4578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:42:04.389969 4578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:42:04.391604 4578 catalog_manager.cc:1259] Loaded cluster ID: a9b194bc98234cb489917d9c775cf297
I20260806 22:42:04.391736 4578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:42:04.395802 4578 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:42:04.397399 4578 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a061e50892684eef8975caba1a8f697d: Loaded TSK: 0
I20260806 22:42:04.398952 4578 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 22:42:04.660490 4559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:04.660825 4559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:04.660995 4559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:04.668663 4559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:04.668812 4559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:04.668998 4559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:42:04.679435 4559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.129.keytab
--principal=oryx/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:34519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=35785
--tserver_master_addrs=127.23.100.190:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:04.680961 4559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:04.682297 4559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:04.697971 4613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:04.698437 4614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:04.698829 4617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:04.699909 4559 server_base.cc:1034] running on GCE node
I20260806 22:42:04.700608 4559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:04.701700 4559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:04.702930 4559 hybrid_clock.cc:648] HybridClock initialized: now 1786056124702886 us; error 42 us; skew 500 ppm
I20260806 22:42:04.703269 4559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:04 dist-test-slave-zr1t krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056124, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:04.706624 4559 init.cc:377] Logged in from keytab as oryx/127.23.100.129@KRBTEST.COM (short username oryx)
I20260806 22:42:04.708305 4559 webserver.cc:460] Webserver started at http://127.23.100.129:35785/ using document root <none> and password file <none>
I20260806 22:42:04.708995 4559 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:04.709139 4559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:04.714931 4559 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 22:42:04.717878 4626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:04.719358 4559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:42:04.719564 4559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "f9970751ccc4410cb400f0f7e5da02c1"
format_stamp: "Formatted at 2026-08-06 22:42:01 on dist-test-slave-zr1t"
server_key: "67207ff8b66170ba36418483c6e893cc"
server_key_iv: "acd047860c74006e99a9449eec674313"
server_key_version: "encryptionkey@0"
I20260806 22:42:04.720530 4559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:04.733388 4559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:04.737892 4559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:04.738179 4559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:04.739233 4559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:04.740724 4559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:04.740823 4559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:04.740975 4559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:04.741077 4559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:04.782941 4559 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:34519
I20260806 22:42:04.783098 4800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:34519 every 8 connection(s)
I20260806 22:42:04.784741 4559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:42:04.789892 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 4559
I20260806 22:42:04.790587 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:40839
--local_ip_for_outbound_sockets=127.23.100.130
--tserver_master_addrs=127.23.100.190:33557
--webserver_port=37993
--webserver_interface=127.23.100.130
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.100.130
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:42:04 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:04 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:04.807862 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:04.788161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:57885)
0806 22:42:04.789099 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:04.789149 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:04.791342 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:04.793225 (+ 1883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:04.793233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:04.793899 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:04.794481 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:04.794501 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.797172 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.797178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:04.798699 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:04.798715 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.800457 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.800463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:04.800681 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:04.801509 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:04.801609 (+ 100us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:04.806262 (+ 4653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":402,"threads_started":1}
W20260806 22:42:04.807962 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:04.788174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:57885 (local address 127.23.100.190:33557)
0806 22:42:04.788954 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:04.788968 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:04.789303 (+ 335us) server_negotiation.cc:407] Connection header received
0806 22:42:04.792055 (+ 2752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:04.792081 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:04.792631 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:04.793088 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:04.794732 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.796939 (+ 2207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.798886 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.800179 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.800425 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:04.806736 (+ 6311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:57885: BlockingRecv error: recv got EOF from 127.23.100.129:57885 (error 108)
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":324,"threads_started":1}
W20260806 22:42:04.809074 4803 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:33557 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.100.190:33557: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Aug 06 22:42:04 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:04 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:04.819897 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:04.809859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:45359)
0806 22:42:04.810106 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:04.810128 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:04.810268 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:04.810724 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:04.810728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:04.811120 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:04.811283 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:04.811292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.812948 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.812952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:04.814236 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:04.814252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.815360 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.815365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:04.815514 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:04.816246 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:04.816266 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:04.819642 (+ 3376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260806 22:42:04.820290 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:04.810113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:45359 (local address 127.23.100.190:33557)
0806 22:42:04.810343 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:04.810350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:04.810374 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:42:04.810442 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:04.810450 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:04.810539 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:04.810656 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:04.811453 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.812754 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.814411 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.815194 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.815345 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:04.819985 (+ 4640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:45359: BlockingRecv error: recv got EOF from 127.23.100.129:45359 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 22:42:04 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:04 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:04.830096 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:04.820799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:45213)
0806 22:42:04.821016 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:04.821061 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:04.821180 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:04.821572 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:04.821578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:04.821944 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:04.822105 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:04.822114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.823249 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.823257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:04.824371 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:04.824386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.825190 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.825196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:04.825432 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:04.826135 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:04.826154 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:04.829732 (+ 3578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 22:42:04.830437 4803 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 22:42:04.830580 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:04.820983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:45213 (local address 127.23.100.190:33557)
0806 22:42:04.821183 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:04.821190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:04.821212 (+ 22us) server_negotiation.cc:407] Connection header received
0806 22:42:04.821286 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:04.821292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:04.821394 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:04.821511 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:04.822283 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.823088 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.824554 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:04.825018 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:04.825183 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:04.829975 (+ 4792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:45213: BlockingRecv error: recv got EOF from 127.23.100.129:45213 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 22:42:05.149737 4809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:05.150099 4809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:05.150269 4809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:05.157776 4809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:05.157938 4809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:05.158144 4809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:42:05.168346 4809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.130.keytab
--principal=oryx/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:40839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=37993
--tserver_master_addrs=127.23.100.190:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:05.169886 4809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:05.171259 4809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:05.184769 4823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:05.184876 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:05.184739 4825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:05.186120 4809 server_base.cc:1034] running on GCE node
I20260806 22:42:05.186949 4809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:05.188040 4809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:05.189342 4809 hybrid_clock.cc:648] HybridClock initialized: now 1786056125189299 us; error 64 us; skew 500 ppm
I20260806 22:42:05.189776 4809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056125, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:05.193128 4809 init.cc:377] Logged in from keytab as oryx/127.23.100.130@KRBTEST.COM (short username oryx)
I20260806 22:42:05.194715 4809 webserver.cc:460] Webserver started at http://127.23.100.130:37993/ using document root <none> and password file <none>
I20260806 22:42:05.195387 4809 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:05.195503 4809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:05.200941 4809 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 22:42:05.203857 4834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:05.205260 4809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:42:05.205439 4809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "23bacee2e94a41929d08be1f48f03681"
format_stamp: "Formatted at 2026-08-06 22:42:02 on dist-test-slave-zr1t"
server_key: "da1c5a7fadf4f6f678ffe22c56be14c1"
server_key_iv: "1b0ebb05b6c868868ed831fe9bbc293e"
server_key_version: "encryptionkey@0"
I20260806 22:42:05.206501 4809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:05.221241 4809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:05.226146 4809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:05.226406 4809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:05.227365 4809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:05.228747 4809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:05.228845 4809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:05.228930 4809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:05.228992 4809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:05.265717 4809 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:40839
I20260806 22:42:05.265771 5025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:40839 every 8 connection(s)
I20260806 22:42:05.267253 4809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:42:05.276821 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 4809
I20260806 22:42:05.277626 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:45347
--local_ip_for_outbound_sockets=127.23.100.131
--tserver_master_addrs=127.23.100.190:33557
--webserver_port=43523
--webserver_interface=127.23.100.131
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.100.131
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 22:42:05.284636 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:05.270087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:55147 (local address 127.23.100.190:33557)
0806 22:42:05.270262 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:05.270267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:05.271491 (+ 1224us) server_negotiation.cc:407] Connection header received
0806 22:42:05.273355 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:05.273361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:05.273453 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:05.273597 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:05.275098 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.275861 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.278201 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.278695 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.278881 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.284382 (+ 5501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:55147: BlockingRecv error: recv got EOF from 127.23.100.130:55147 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 22:42:05.285387 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:05.270582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:55147)
0806 22:42:05.271312 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:05.271353 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:05.273076 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:05.273826 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:05.273833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:05.274351 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:05.274913 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.274931 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.276051 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.276055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.278017 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.278037 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.279004 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.279013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.279270 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.280162 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:05.280187 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:05.284183 (+ 3996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":291,"threads_started":1}
W20260806 22:42:05.286197 5026 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:33557 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.100.190:33557: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:05.295662 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:05.286949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:36967)
0806 22:42:05.287169 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:05.287190 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:05.287326 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:05.287703 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:05.287708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:05.288083 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:05.288243 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.288251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.289378 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.289384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.290444 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.290457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.291258 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.291276 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.291465 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.292191 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:05.292210 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:05.295343 (+ 3133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260806 22:42:05.295883 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:05.287074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:36967 (local address 127.23.100.190:33557)
0806 22:42:05.287263 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:05.287269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:05.287290 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:42:05.287476 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:05.287482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:05.287558 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:05.287672 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:05.288411 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.289220 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.290624 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.291079 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.291232 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.295579 (+ 4347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:36967: BlockingRecv error: recv got EOF from 127.23.100.130:36967 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:05.304910 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:05.296708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:37953)
0806 22:42:05.296977 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:05.296996 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:05.297101 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:05.297462 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:05.297467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:05.297803 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:05.297951 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.297960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.299015 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.299025 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.299963 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.299975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.300766 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.300772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.300982 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.301651 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:05.301668 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:05.304598 (+ 2930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150}
W20260806 22:42:05.305073 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:05.296908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:37953 (local address 127.23.100.190:33557)
0806 22:42:05.297073 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:05.297078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:05.297097 (+ 19us) server_negotiation.cc:407] Connection header received
0806 22:42:05.297246 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:05.297251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:05.297326 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:05.297434 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:05.298101 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.298859 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.300109 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.300614 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.300753 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.304825 (+ 4072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:37953: BlockingRecv error: recv got EOF from 127.23.100.130:37953 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 22:42:05.305502 5026 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 22:42:05.627458 5031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:05.627729 5031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:05.627825 5031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:05.635028 5031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:05.635134 5031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:05.635255 5031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:42:05.645018 5031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/krb5kdc/oryx_127.23.100.131.keytab
--principal=oryx/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:45347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=43523
--tserver_master_addrs=127.23.100.190:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:05.646332 5031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:05.647568 5031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:05.660921 5049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:05.660965 5042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:05.660925 5040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:05.661865 5031 server_base.cc:1034] running on GCE node
I20260806 22:42:05.662571 5031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:05.663542 5031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:05.664783 5031 hybrid_clock.cc:648] HybridClock initialized: now 1786056125664740 us; error 60 us; skew 500 ppm
I20260806 22:42:05.665127 5031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056125, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:05.668432 5031 init.cc:377] Logged in from keytab as oryx/127.23.100.131@KRBTEST.COM (short username oryx)
I20260806 22:42:05.670084 5031 webserver.cc:460] Webserver started at http://127.23.100.131:43523/ using document root <none> and password file <none>
I20260806 22:42:05.670730 5031 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:05.670847 5031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:05.676301 5031 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 22:42:05.678952 5066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:05.680419 5031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 22:42:05.680553 5031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "e0721e4ecc70479190c0334b53daa937"
format_stamp: "Formatted at 2026-08-06 22:42:03 on dist-test-slave-zr1t"
server_key: "ffa4e8c882b187396ca06d7175818893"
server_key_iv: "13a2b40581cddb92b2d8fb10193569e1"
server_key_version: "encryptionkey@0"
I20260806 22:42:05.681460 5031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:05.700891 5031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:05.705060 5031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:05.705294 5031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:05.706303 5031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:05.707579 5031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:05.707667 5031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:05.707751 5031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:05.707805 5031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:05.743378 5031 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:45347
I20260806 22:42:05.743546 5239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:45347 every 8 connection(s)
I20260806 22:42:05.744957 5031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:42:05.754287 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 5031
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 22:42:05.761349 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:05.747635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:58893 (local address 127.23.100.190:33557)
0806 22:42:05.747852 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:05.747857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:05.749079 (+ 1222us) server_negotiation.cc:407] Connection header received
0806 22:42:05.750964 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:05.750969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:05.751039 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:05.751132 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:05.752666 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.753411 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.755088 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.755787 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.755970 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.761082 (+ 5112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:58893: BlockingRecv error: recv got EOF from 127.23.100.131:58893 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 22:42:05.762259 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:05.748206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:58893)
0806 22:42:05.748919 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:05.748956 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:05.750695 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:05.751402 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:05.751410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:05.751938 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:05.752495 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.752512 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.753621 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.753625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.754938 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.754950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.756226 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.756232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.756473 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.757173 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:05.757200 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:05.760853 (+ 3653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":278,"threads_started":1}
W20260806 22:42:05.763055 5240 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:33557 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.100.190:33557: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:05.772150 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:05.755397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35728 (local address 127.23.100.190:33557)
0806 22:42:05.756225 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:05.756234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:05.756250 (+ 16us) server_negotiation.cc:407] Connection header received
0806 22:42:05.756300 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:05.756303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:05.756355 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:05.756476 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:05.758099 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.758973 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.760323 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.760764 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.760903 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.764364 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:05.764379 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:05.764401 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:05.764460 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:05.768258 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:05.769207 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:05.769216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:05.769226 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:05.769478 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:05.769873 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:05.769880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:05.769884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:05.770537 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:05.770881 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:05.771339 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 22:42:05.771706 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":571,"threads_started":1}
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:05.773720 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:05.763698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:33539)
0806 22:42:05.763875 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:05.763894 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:05.763997 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:05.764278 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:05.764283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:05.764717 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:05.764909 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.764924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.766078 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.766084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.767535 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.767551 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.768380 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.768388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.768613 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.769204 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:05.769227 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:05.773399 (+ 4172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 22:42:05.774219 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:05.763811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:33539 (local address 127.23.100.190:33557)
0806 22:42:05.763958 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:05.763962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:05.763978 (+ 16us) server_negotiation.cc:407] Connection header received
0806 22:42:05.764106 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:05.764110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:05.764163 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:05.764253 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:05.765067 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.765927 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.767752 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.768219 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.768388 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.773640 (+ 5252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:33539: BlockingRecv error: recv got EOF from 127.23.100.131:33539 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:05.782666 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:05.774414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:57093)
0806 22:42:05.774772 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:05.774794 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:05.774912 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:05.775209 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:05.775214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:05.775590 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:05.775790 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.775803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.776866 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.776870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.777871 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.777882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.778643 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.778647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.778802 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.779482 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:05.779500 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:05.782460 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 22:42:05.783003 5240 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 22:42:05.783274 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:05.774683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:57093 (local address 127.23.100.190:33557)
0806 22:42:05.774897 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:05.774901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:05.774916 (+ 15us) server_negotiation.cc:407] Connection header received
0806 22:42:05.775020 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:05.775023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:05.775073 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:05.775180 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:05.775951 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.776734 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.778066 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.778483 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.778591 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.782701 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:57093: BlockingRecv error: recv got EOF from 127.23.100.131:57093 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:05 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:05.840683 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:05.831125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:49571)
0806 22:42:05.831659 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:05.831677 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:05.831803 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:05.832121 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:05.832124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:05.832631 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:05.832835 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.832845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.834063 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.834067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.835082 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:05.835095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.835920 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.835930 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:05.836072 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.837074 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:05.837108 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:05.840439 (+ 3331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":313,"threads_started":1}
W20260806 22:42:05.840971 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:05.831223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:49571 (local address 127.23.100.190:33557)
0806 22:42:05.831467 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:05.831474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:05.831764 (+ 290us) server_negotiation.cc:407] Connection header received
0806 22:42:05.831909 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:05.831913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:05.831977 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:05.832063 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:05.833029 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.833910 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.835228 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:05.835697 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:05.835800 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:05.840738 (+ 4938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:49571: BlockingRecv error: recv got EOF from 127.23.100.129:49571 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 22:42:06 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:06 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:06.315658 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:06.306176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:36895)
0806 22:42:06.306705 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:06.306722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:06.306848 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:06.307226 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:06.307230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:06.307730 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:06.307948 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:06.307958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.309497 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.309502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:06.310695 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:06.310710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.311535 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.311541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:06.311677 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:06.312272 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:06.312290 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:06.315412 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":300,"threads_started":1}
W20260806 22:42:06.315811 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:06.306368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:36895 (local address 127.23.100.190:33557)
0806 22:42:06.306625 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:06.306632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:06.306803 (+ 171us) server_negotiation.cc:407] Connection header received
0806 22:42:06.306967 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:06.306971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:06.307036 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:06.307122 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:06.308147 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.309281 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.310882 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.311329 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.311430 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:06.315588 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:36895: BlockingRecv error: recv got EOF from 127.23.100.130:36895 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 22:42:06 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:06 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:06.792511 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:06.783686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:56441)
0806 22:42:06.784234 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:06.784250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:06.784362 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:06.784711 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:06.784733 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:06.785265 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:06.785472 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:06.785482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.786801 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.786805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:06.787866 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:06.787879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.788613 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.788618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:06.788744 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:06.789404 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:06.789422 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:06.792287 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":308,"threads_started":1}
W20260806 22:42:06.792757 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:06.783718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:56441 (local address 127.23.100.190:33557)
0806 22:42:06.783891 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:06.783895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:06.784347 (+ 452us) server_negotiation.cc:407] Connection header received
0806 22:42:06.784465 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:06.784471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:06.784558 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:06.784667 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:06.785717 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.786664 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.788025 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.788452 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.788557 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:06.792515 (+ 3958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:56441: BlockingRecv error: recv got EOF from 127.23.100.131:56441 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 22:42:06 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:06 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:06.850821 5251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:06.841918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:36253)
0806 22:42:06.842537 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:06.842560 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:06.842698 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:06.843138 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:06.843143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:06.843698 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:06.843966 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:06.843981 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.845100 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.845104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:06.846129 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:06.846142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.846878 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.846882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:06.847007 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:06.847618 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:06.847635 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:06.850563 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":329,"threads_started":1}
W20260806 22:42:06.851128 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:06.841990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:36253 (local address 127.23.100.190:33557)
0806 22:42:06.842268 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:06.842276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:06.842658 (+ 382us) server_negotiation.cc:407] Connection header received
0806 22:42:06.842814 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:06.842819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:06.842927 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:06.843043 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:06.844188 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.844970 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.846270 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:06.846724 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:06.846832 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:06.850752 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:36253: BlockingRecv error: recv got EOF from 127.23.100.129:36253 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 06 22:42:07 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:07 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:07.325947 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:07.316769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:37821)
0806 22:42:07.317375 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:07.317392 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:07.317593 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:07.318093 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:07.318100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:07.318647 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:07.318875 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:07.318889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.320039 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.320044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:07.320917 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:07.320928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.321654 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.321658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:07.321770 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:07.322405 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:07.322426 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:07.325701 (+ 3275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":332,"threads_started":1}
W20260806 22:42:07.326195 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:07.316983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:37821 (local address 127.23.100.190:33557)
0806 22:42:07.317205 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:07.317210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:07.317500 (+ 290us) server_negotiation.cc:407] Connection header received
0806 22:42:07.317788 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:07.317793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:07.317924 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:07.318032 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:07.319071 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.319906 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.321058 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.321457 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.321633 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:07.325904 (+ 4271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:37821: BlockingRecv error: recv got EOF from 127.23.100.130:37821 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 22:42:07 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:07 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:07.802764 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:07.793476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:38455)
0806 22:42:07.794072 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:07.794088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:07.794244 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:07.794649 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:07.794654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:07.795102 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:07.795311 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:07.795321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.796431 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.796437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:07.797343 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:07.797355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.798155 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.798159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:07.798282 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:07.798860 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:07.798878 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:07.802402 (+ 3524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":295,"threads_started":1}
W20260806 22:42:07.802839 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:07.793684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:38455 (local address 127.23.100.190:33557)
0806 22:42:07.793927 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:07.793934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:07.794176 (+ 242us) server_negotiation.cc:407] Connection header received
0806 22:42:07.794408 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:07.794413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:07.794505 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:07.794613 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:07.795465 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.796278 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.797489 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.797969 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.798128 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:07.802572 (+ 4444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:38455: BlockingRecv error: recv got EOF from 127.23.100.131:38455 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 06 22:42:07 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:07 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:07.859838 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:07.851788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:51293)
0806 22:42:07.852257 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:07.852273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:07.852396 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:07.852694 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:07.852697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:07.853061 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:07.853228 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:07.853237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.854391 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.854395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:07.855281 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:07.855293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.855947 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.855962 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:07.856083 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:07.856712 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:07.856730 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:07.859564 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":275,"threads_started":1}
W20260806 22:42:07.860025 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:07.851818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:51293 (local address 127.23.100.190:33557)
0806 22:42:07.852051 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:07.852061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:07.852360 (+ 299us) server_negotiation.cc:407] Connection header received
0806 22:42:07.852497 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:07.852501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:07.852564 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:07.852647 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:07.853398 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.854262 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.855423 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:07.855813 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:07.855919 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:07.859777 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:51293: BlockingRecv error: recv got EOF from 127.23.100.129:51293 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 22:42:08 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:08 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:08.335736 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:08.327029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:35085)
0806 22:42:08.327566 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:08.327585 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:08.327718 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:08.328060 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:08.328064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:08.328476 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:08.328668 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:08.328677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.329885 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.329889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:08.330787 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:08.330800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.331591 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.331598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:08.331801 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:08.332458 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:08.332476 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:08.335362 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":316,"threads_started":1}
W20260806 22:42:08.335879 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:08.327113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:35085 (local address 127.23.100.190:33557)
0806 22:42:08.327389 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:08.327397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:08.327706 (+ 309us) server_negotiation.cc:407] Connection header received
0806 22:42:08.327841 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:08.327845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:08.327912 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:08.328003 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:08.328829 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.329709 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.330941 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.331427 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.331582 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:08.335618 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:35085: BlockingRecv error: recv got EOF from 127.23.100.130:35085 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 22:42:08 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:08 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:08.813187 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:08.803843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:33295)
0806 22:42:08.804431 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:08.804453 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:08.804589 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:08.805014 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:08.805018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:08.805376 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:08.805656 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:08.805670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.806853 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.806858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:08.807918 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:08.807933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.808712 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.808717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:08.808874 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:08.809606 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:08.809625 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:08.812870 (+ 3245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":370,"threads_started":1}
W20260806 22:42:08.813576 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:08.804032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:33295 (local address 127.23.100.190:33557)
0806 22:42:08.804217 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:08.804222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:08.804585 (+ 363us) server_negotiation.cc:407] Connection header received
0806 22:42:08.804718 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:08.804722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:08.804820 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:08.804928 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:08.805846 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.806652 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.808088 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.808547 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.808654 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:08.813286 (+ 4632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:33295: BlockingRecv error: recv got EOF from 127.23.100.131:33295 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 22:42:08 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:08 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:08.870460 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:08.860921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:53761)
0806 22:42:08.861567 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:08.861594 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:08.861755 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:08.862142 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:08.862147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:08.862583 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:08.862787 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:08.862801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.864079 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.864088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:08.865324 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:08.865340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.866143 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.866149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:08.866351 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:08.867069 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:08.867089 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:08.870205 (+ 3116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":303,"threads_started":1}
W20260806 22:42:08.870764 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:08.861105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:53761 (local address 127.23.100.190:33557)
0806 22:42:08.861389 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:08.861396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:08.861680 (+ 284us) server_negotiation.cc:407] Connection header received
0806 22:42:08.861918 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:08.861925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:08.861996 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:08.862107 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:08.862964 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.863914 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.865488 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:08.865965 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:08.866114 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:08.870434 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:53761: BlockingRecv error: recv got EOF from 127.23.100.129:53761 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 06 22:42:09 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:09 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:09.348423 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:09.336841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:53427)
0806 22:42:09.337428 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:09.337460 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:09.337640 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:09.338077 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:09.338083 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:09.338624 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:09.338780 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:09.338789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.339875 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.339879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:09.340848 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:09.340860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.341616 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.341620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:09.341741 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:09.345313 (+ 3572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:09.345332 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:09.348200 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":299,"threads_started":1}
W20260806 22:42:09.348641 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:09.336858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:53427 (local address 127.23.100.190:33557)
0806 22:42:09.337055 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:09.337061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:09.337595 (+ 534us) server_negotiation.cc:407] Connection header received
0806 22:42:09.337824 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:09.337832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:09.337921 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:09.338002 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:09.338965 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.339755 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.340984 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.341401 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.341554 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:09.348406 (+ 6852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:53427: BlockingRecv error: recv got EOF from 127.23.100.130:53427 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 06 22:42:09 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:09 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:09.822836 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:09.814197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:42859)
0806 22:42:09.814764 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:09.814781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:09.814902 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:09.815266 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:09.815269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:09.815653 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:09.815805 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:09.815814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.816988 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.816991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:09.818060 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:09.818073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.818794 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.818799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:09.818925 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:09.819644 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:09.819662 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:09.822512 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":326,"threads_started":1}
W20260806 22:42:09.823037 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:09.814315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:42859 (local address 127.23.100.190:33557)
0806 22:42:09.814603 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:09.814608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:09.814868 (+ 260us) server_negotiation.cc:407] Connection header received
0806 22:42:09.815052 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:09.815056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:09.815124 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:09.815217 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:09.815962 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.816857 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.818203 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.818647 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.818750 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:09.822754 (+ 4004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:42859: BlockingRecv error: recv got EOF from 127.23.100.131:42859 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 22:42:09 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:09 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:09.880796 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:09.871462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:44707)
0806 22:42:09.872073 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:09.872094 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:09.872245 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:09.872753 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:09.872759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:09.873401 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:09.873675 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:09.873691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.874915 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.874919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:09.875901 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:09.875914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.876629 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.876634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:09.876780 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:09.877467 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:09.877485 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:09.880477 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":300,"threads_started":1}
W20260806 22:42:09.880935 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:09.871597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:44707 (local address 127.23.100.190:33557)
0806 22:42:09.871846 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:09.871853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:09.872189 (+ 336us) server_negotiation.cc:407] Connection header received
0806 22:42:09.872402 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:09.872408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:09.872495 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:09.872599 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:09.873848 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.874770 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.876036 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:09.876431 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:09.876546 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:09.880728 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:44707: BlockingRecv error: recv got EOF from 127.23.100.129:44707 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 22:42:10 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:10 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:10.358363 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:10.349612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:59173)
0806 22:42:10.350209 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:10.350227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:10.350349 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:10.350747 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:10.350751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:10.351223 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:10.351439 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:10.351448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.352473 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.352477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:10.353409 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:10.353422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.354198 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.354205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:10.354391 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:10.355072 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:10.355091 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:10.358007 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":295,"threads_started":1}
W20260806 22:42:10.358500 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:10.349943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:59173 (local address 127.23.100.190:33557)
0806 22:42:10.350148 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:10.350154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:10.350335 (+ 181us) server_negotiation.cc:407] Connection header received
0806 22:42:10.350508 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:10.350512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:10.350576 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:10.350661 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:10.351578 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.352331 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.353554 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.354035 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.354176 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:10.358212 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:59173: BlockingRecv error: recv got EOF from 127.23.100.130:59173 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 22:42:10 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:10 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:10.833635 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:10.823984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:55919)
0806 22:42:10.824575 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:10.824592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:10.824781 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:10.825186 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:10.825190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:10.825707 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:10.825901 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:10.825910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.827037 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.827042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:10.828010 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:10.828024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.828809 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.828814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:10.828958 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:10.829744 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:10.829778 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:10.833289 (+ 3511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":306,"threads_started":1}
W20260806 22:42:10.833884 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:10.824092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:55919 (local address 127.23.100.190:33557)
0806 22:42:10.824302 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:10.824307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:10.824717 (+ 410us) server_negotiation.cc:407] Connection header received
0806 22:42:10.824889 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:10.824892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:10.824961 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:10.825101 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:10.826074 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.826858 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.828183 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.828646 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.828758 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:10.833575 (+ 4817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:55919: BlockingRecv error: recv got EOF from 127.23.100.131:55919 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 22:42:10 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:10 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:10.890540 5265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:10.881881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:58707)
0806 22:42:10.882454 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:10.882471 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:10.882588 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:10.882895 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:10.882898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:10.883291 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:10.883472 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:10.883482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.884656 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.884667 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:10.885639 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:10.885652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.886478 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.886483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:10.886649 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:10.887285 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:10.887332 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:10.890210 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":313,"threads_started":1}
W20260806 22:42:10.891024 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:10.881926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:58707 (local address 127.23.100.190:33557)
0806 22:42:10.882111 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:10.882116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:10.882548 (+ 432us) server_negotiation.cc:407] Connection header received
0806 22:42:10.882686 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:10.882690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:10.882756 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:10.882846 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:10.883602 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.884427 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.885799 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:10.886302 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:10.886404 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:10.890501 (+ 4097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:58707: BlockingRecv error: recv got EOF from 127.23.100.129:58707 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 22:42:11 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:11 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:11.368777 5267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:11.359361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:44593)
0806 22:42:11.359992 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:11.360009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:11.360178 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:11.360564 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:11.360569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:11.361014 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:11.361190 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:11.361199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.362317 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.362323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:11.363232 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:11.363244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.364011 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.364017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:11.364198 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:11.364846 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:11.364894 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:11.368446 (+ 3552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":352,"threads_started":1}
W20260806 22:42:11.369088 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:11.359545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:44593 (local address 127.23.100.190:33557)
0806 22:42:11.359809 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:11.359816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:11.360104 (+ 288us) server_negotiation.cc:407] Connection header received
0806 22:42:11.360316 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:11.360326 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:11.360414 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:11.360526 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:11.361349 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.362156 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.363396 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.363831 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.363968 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:11.368794 (+ 4826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:44593: BlockingRecv error: recv got EOF from 127.23.100.130:44593 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 06 22:42:11 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:11 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:11.843452 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:11.834674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:50679)
0806 22:42:11.835278 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:11.835295 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:11.835415 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:11.835723 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:11.835727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:11.836176 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:11.836340 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:11.836350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.837420 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.837424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:11.838450 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:11.838464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.839171 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.839176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:11.839308 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:11.840031 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:11.840052 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:11.843142 (+ 3090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":350,"threads_started":1}
W20260806 22:42:11.843654 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:11.834678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:50679 (local address 127.23.100.190:33557)
0806 22:42:11.834903 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:11.834912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:11.835411 (+ 499us) server_negotiation.cc:407] Connection header received
0806 22:42:11.835515 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:11.835519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:11.835584 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:11.835664 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:11.836510 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.837278 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.838603 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.839026 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.839125 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:11.843352 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:50679: BlockingRecv error: recv got EOF from 127.23.100.131:50679 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 22:42:11 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:11 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:11.900636 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:11.891671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:39553)
0806 22:42:11.892298 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:11.892317 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:11.892481 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:11.892820 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:11.892823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:11.893291 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:11.893500 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:11.893510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.894768 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.894772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:11.895703 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:11.895715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.896497 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.896504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:11.896674 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:11.897255 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:11.897273 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:11.900342 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":321,"threads_started":1}
W20260806 22:42:11.900765 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:11.891780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:39553 (local address 127.23.100.190:33557)
0806 22:42:11.892020 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:11.892030 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:11.892409 (+ 379us) server_negotiation.cc:407] Connection header received
0806 22:42:11.892607 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:11.892611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:11.892680 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:11.892768 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:11.893716 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.894585 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.895864 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:11.896324 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:11.896474 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:11.900591 (+ 4117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:39553: BlockingRecv error: recv got EOF from 127.23.100.129:39553 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 22:42:12 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:12 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:12.379948 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:12.369927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:43689)
0806 22:42:12.370793 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:12.370815 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:12.370981 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:12.371447 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:12.371451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:12.372101 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:12.372384 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:12.372403 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.373492 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.373496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:12.374461 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:12.374473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.375154 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.375159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:12.375276 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:12.375927 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:12.375974 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:12.379576 (+ 3602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":479,"threads_started":1}
W20260806 22:42:12.380084 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:12.369972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:43689 (local address 127.23.100.190:33557)
0806 22:42:12.370266 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:12.370276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:12.370914 (+ 638us) server_negotiation.cc:407] Connection header received
0806 22:42:12.371133 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:12.371138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:12.371265 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:12.371402 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:12.372558 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.373330 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.374597 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.375015 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.375110 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:12.379798 (+ 4688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:43689: BlockingRecv error: recv got EOF from 127.23.100.130:43689 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Aug 06 22:42:12 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:12 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:12.853483 5274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:12.844511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:56045)
0806 22:42:12.845213 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:12.845231 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:12.845356 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:12.845696 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:12.845700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:12.846283 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:12.846494 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:12.846505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.847550 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.847555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:12.848542 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:12.848556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.849354 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.849366 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:12.849488 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:12.850183 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:12.850202 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:12.853226 (+ 3024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":322,"threads_started":1}
W20260806 22:42:12.853807 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:12.844606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:56045 (local address 127.23.100.190:33557)
0806 22:42:12.844770 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:12.844774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:12.845307 (+ 533us) server_negotiation.cc:407] Connection header received
0806 22:42:12.845471 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:12.845475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:12.845562 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:12.845680 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:12.846661 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.847382 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.848702 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.849146 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.849252 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:12.853499 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:56045: BlockingRecv error: recv got EOF from 127.23.100.131:56045 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 22:42:12 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:12 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:12.911091 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:12.901835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:55953)
0806 22:42:12.902440 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:12.902456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:12.902579 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:12.902915 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:12.902918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:12.903333 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:12.903490 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:12.903498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.904923 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.904931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:12.906177 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:12.906192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.907057 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.907062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:12.907179 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:12.907838 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:12.907856 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:12.910840 (+ 2984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":297,"threads_started":1}
W20260806 22:42:12.911525 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:12.902085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:55953 (local address 127.23.100.190:33557)
0806 22:42:12.902281 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:12.902285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:12.902539 (+ 254us) server_negotiation.cc:407] Connection header received
0806 22:42:12.902693 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:12.902696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:12.902760 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:12.902852 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:12.903654 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.904710 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.906333 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:12.906882 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:12.906992 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:12.911235 (+ 4243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:55953: BlockingRecv error: recv got EOF from 127.23.100.129:55953 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 22:42:13 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:13 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:13.390269 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:13.381048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:37119)
0806 22:42:13.381610 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:13.381629 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:13.381757 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:13.382057 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:13.382061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:13.382519 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:13.382689 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:13.382699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.383881 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.383886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:13.385024 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:13.385040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.385858 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.385864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:13.385989 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:13.386749 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:13.386769 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:13.389887 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":267,"threads_started":1}
W20260806 22:42:13.390372 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:13.381244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:37119 (local address 127.23.100.190:33557)
0806 22:42:13.381482 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:13.381491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:13.381709 (+ 218us) server_negotiation.cc:407] Connection header received
0806 22:42:13.381858 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:13.381862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:13.381922 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:13.382007 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:13.382820 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.383713 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.385164 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.385656 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.385772 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:13.390088 (+ 4316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:37119: BlockingRecv error: recv got EOF from 127.23.100.130:37119 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 06 22:42:13 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:13 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:13.863528 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:13.854693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:49129)
0806 22:42:13.855309 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:13.855355 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:13.855544 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:13.855946 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:13.855952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:13.856514 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:13.856730 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:13.856743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.857939 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.857943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:13.858877 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:13.858888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.859657 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.859662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:13.859775 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:13.860465 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:13.860484 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:13.863285 (+ 2801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":359,"threads_started":1}
W20260806 22:42:13.863824 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:13.854854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:49129 (local address 127.23.100.190:33557)
0806 22:42:13.855115 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:13.855122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:13.855477 (+ 355us) server_negotiation.cc:407] Connection header received
0806 22:42:13.855691 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:13.855697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:13.855788 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:13.855874 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:13.856941 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.857788 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.859046 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.859492 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.859599 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:13.863535 (+ 3936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:49129: BlockingRecv error: recv got EOF from 127.23.100.131:49129 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 22:42:13 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:13 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:13.921375 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:13.912136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:42403)
0806 22:42:13.912662 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:13.912679 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:13.912844 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:13.913202 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:13.913207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:13.913712 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:13.913896 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:13.913907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.914963 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.914975 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:13.916066 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:13.916080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.916967 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.916974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:13.917154 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:13.917923 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:13.917942 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:13.921123 (+ 3181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":278,"threads_started":1}
W20260806 22:42:13.921701 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:13.912212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:42403 (local address 127.23.100.190:33557)
0806 22:42:13.912467 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:13.912474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:13.912769 (+ 295us) server_negotiation.cc:407] Connection header received
0806 22:42:13.912983 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:13.912992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:13.913056 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:13.913164 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:13.914046 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.914808 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.916217 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:13.916812 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:13.916948 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:13.921399 (+ 4451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:42403: BlockingRecv error: recv got EOF from 127.23.100.129:42403 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 22:42:14 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:14 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:14.406862 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:14.391357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:33187)
0806 22:42:14.391949 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:14.391965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:14.392082 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:14.392487 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:14.392490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:14.392934 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:14.393150 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:14.393159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.395002 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.395008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:14.396178 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:14.396195 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.397433 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.397438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:14.397642 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:14.402104 (+ 4462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:14.402143 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:14.406594 (+ 4451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":310,"threads_started":1}
W20260806 22:42:14.407380 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:14.391443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:33187 (local address 127.23.100.190:33557)
0806 22:42:14.391685 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:14.391694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:14.392054 (+ 360us) server_negotiation.cc:407] Connection header received
0806 22:42:14.392210 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:14.392218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:14.392320 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:14.392457 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:14.393310 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.394814 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.396339 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.397257 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.397431 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:14.406969 (+ 9538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:33187: BlockingRecv error: recv got EOF from 127.23.100.130:33187 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 22:42:14 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:14 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:14.873550 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:14.864519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:56677)
0806 22:42:14.865174 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:14.865191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:14.865309 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:14.865621 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:14.865624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:14.866040 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:14.866219 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:14.866229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.867245 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.867249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:14.868264 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:14.868276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.869044 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.869048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:14.869170 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:14.869912 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:14.869930 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:14.873192 (+ 3262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":377,"threads_started":1}
W20260806 22:42:14.873773 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:14.864658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:56677 (local address 127.23.100.190:33557)
0806 22:42:14.865001 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:14.865005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:14.865319 (+ 314us) server_negotiation.cc:407] Connection header received
0806 22:42:14.865394 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:14.865397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:14.865474 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:14.865585 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:14.866367 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.867111 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.868399 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.868909 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.869005 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:14.873439 (+ 4434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:56677: BlockingRecv error: recv got EOF from 127.23.100.131:56677 (error 108)
Metrics: {"server-negotiator.queue_time_us":203}
Aug 06 22:42:14 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:14 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:14.931437 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:14.922438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:36021)
0806 22:42:14.923011 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:14.923027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:14.923141 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:14.923433 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:14.923436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:14.923889 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:14.924083 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:14.924095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.925311 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.925315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:14.926283 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:14.926296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.927199 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.927203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:14.927322 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:14.928052 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:14.928070 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:14.931168 (+ 3098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":326,"threads_started":1}
W20260806 22:42:14.931792 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:14.922517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:36021 (local address 127.23.100.190:33557)
0806 22:42:14.922748 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:14.922754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:14.923133 (+ 379us) server_negotiation.cc:407] Connection header received
0806 22:42:14.923244 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:14.923248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:14.923314 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:14.923390 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:14.924236 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.925177 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.926451 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:14.927044 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:14.927158 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:14.931439 (+ 4281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:36021: BlockingRecv error: recv got EOF from 127.23.100.129:36021 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 22:42:15 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:15 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:15.416989 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:15.407883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:49855)
0806 22:42:15.408488 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:15.408505 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:15.408674 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:15.409047 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:15.409051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:15.409486 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:15.409666 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:15.409675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.410805 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.410812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:15.411871 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:15.411884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.412614 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.412620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:15.412802 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:15.413542 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:15.413560 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:15.416618 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":336,"threads_started":1}
W20260806 22:42:15.417122 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:15.407995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:49855 (local address 127.23.100.190:33557)
0806 22:42:15.408259 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:15.408267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:15.408602 (+ 335us) server_negotiation.cc:407] Connection header received
0806 22:42:15.408811 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:15.408816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:15.408880 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:15.409016 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:15.409820 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.410670 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.412022 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.412455 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.412594 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:15.416838 (+ 4244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:49855: BlockingRecv error: recv got EOF from 127.23.100.130:49855 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 06 22:42:15 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:15 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:15.889367 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:15.874834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:49777)
0806 22:42:15.875500 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:15.875517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:15.875692 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:15.876141 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:15.876144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:15.876532 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:15.876686 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:15.876695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.878133 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.878139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:15.879424 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:15.879439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.880189 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.880196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:15.880376 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:15.885743 (+ 5367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:15.885768 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:15.889075 (+ 3307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":392,"threads_started":1}
W20260806 22:42:15.889834 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:15.875098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:49777 (local address 127.23.100.190:33557)
0806 22:42:15.875376 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:15.875382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:15.875613 (+ 231us) server_negotiation.cc:407] Connection header received
0806 22:42:15.875897 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:15.875901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:15.875970 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:15.876062 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:15.876860 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.877981 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.879586 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.880021 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.880161 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:15.889437 (+ 9276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:49777: BlockingRecv error: recv got EOF from 127.23.100.131:49777 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 06 22:42:15 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:15 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:15.941403 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:15.932476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:51511)
0806 22:42:15.933109 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:15.933127 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:15.933251 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:15.933597 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:15.933600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:15.934083 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:15.934276 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:15.934286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.935376 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.935382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:15.936312 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:15.936325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.937012 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.937017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:15.937155 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:15.937897 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:15.937918 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:15.941161 (+ 3243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":375,"threads_started":1}
W20260806 22:42:15.941612 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:15.932551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:51511 (local address 127.23.100.190:33557)
0806 22:42:15.932736 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:15.932740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:15.933200 (+ 460us) server_negotiation.cc:407] Connection header received
0806 22:42:15.933350 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:15.933354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:15.933417 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:15.933504 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:15.934424 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.935200 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.936444 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:15.936858 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:15.936958 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:15.941349 (+ 4391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:51511: BlockingRecv error: recv got EOF from 127.23.100.129:51511 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 22:42:16 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:16 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:16.427055 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:16.418140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:43481)
0806 22:42:16.418640 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:16.418656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:16.418815 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:16.419192 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:16.419197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:16.419629 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:16.419819 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:16.419829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.420912 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.420917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:16.421984 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:16.421998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.422862 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.422867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:16.422990 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:16.423613 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:16.423630 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:16.426821 (+ 3191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":213,"threads_started":1}
W20260806 22:42:16.427372 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:16.418286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:43481 (local address 127.23.100.190:33557)
0806 22:42:16.418496 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:16.418502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:16.418745 (+ 243us) server_negotiation.cc:407] Connection header received
0806 22:42:16.418974 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:16.418980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:16.419053 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:16.419158 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:16.419980 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.420722 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.422108 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.422670 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.422790 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:16.427042 (+ 4252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:43481: BlockingRecv error: recv got EOF from 127.23.100.130:43481 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 22:42:16 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:16 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:16.899523 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:16.890620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:59169)
0806 22:42:16.891178 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:16.891198 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:16.891374 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:16.891736 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:16.891740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:16.892169 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:16.892405 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:16.892414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.893584 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.893589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:16.894531 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:16.894549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.895339 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.895344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:16.895514 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:16.896168 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:16.896187 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:16.899263 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":502,"threads_started":1}
W20260806 22:42:16.899791 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:16.890918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:59169 (local address 127.23.100.190:33557)
0806 22:42:16.891115 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:16.891122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:16.891301 (+ 179us) server_negotiation.cc:407] Connection header received
0806 22:42:16.891512 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:16.891516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:16.891586 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:16.891680 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:16.892559 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.893367 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.894711 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.895166 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.895272 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:16.899525 (+ 4253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:59169: BlockingRecv error: recv got EOF from 127.23.100.131:59169 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 22:42:16 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:16 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:16.952312 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:16.942526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:53291)
0806 22:42:16.943184 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:16.943202 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:16.943324 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:16.943805 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:16.943808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:16.944243 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:16.944409 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:16.944419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.945848 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.945853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:16.946908 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:16.946923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.947772 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.947777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:16.947951 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:16.948636 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:16.948655 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:16.951935 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":384,"threads_started":1}
W20260806 22:42:16.952483 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:16.942652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:53291 (local address 127.23.100.190:33557)
0806 22:42:16.942869 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:16.942874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:16.943287 (+ 413us) server_negotiation.cc:407] Connection header received
0806 22:42:16.943540 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:16.943545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:16.943617 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:16.943718 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:16.944665 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.945648 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.947075 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:16.947620 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:16.947730 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:16.952157 (+ 4427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:53291: BlockingRecv error: recv got EOF from 127.23.100.129:53291 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 22:42:17 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:17 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:17.437187 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:17.428116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:36955)
0806 22:42:17.428639 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:17.428655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:17.428779 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:17.429096 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:17.429100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:17.429591 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:17.429760 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:17.429769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.430844 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.430852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:17.432165 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:17.432186 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.433136 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.433142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:17.433272 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:17.433911 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:17.433928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:17.436965 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":319,"threads_started":1}
W20260806 22:42:17.437455 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:17.428170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:36955 (local address 127.23.100.190:33557)
0806 22:42:17.428425 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:17.428435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:17.428765 (+ 330us) server_negotiation.cc:407] Connection header received
0806 22:42:17.428893 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:17.428896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:17.428963 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:17.429054 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:17.429925 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.430663 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.432327 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.432984 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.433084 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:17.437178 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:36955: BlockingRecv error: recv got EOF from 127.23.100.130:36955 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 06 22:42:17 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:17 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:17.909509 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:17.900645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:52807)
0806 22:42:17.901272 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:17.901289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:17.901447 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:17.901872 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:17.901879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:17.902486 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:17.902651 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:17.902661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.903682 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.903686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:17.904608 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:17.904621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.905440 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.905445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:17.905655 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:17.906376 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:17.906395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:17.909268 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":380,"threads_started":1}
W20260806 22:42:17.909899 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:17.900712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:52807 (local address 127.23.100.190:33557)
0806 22:42:17.900976 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:17.900982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:17.901375 (+ 393us) server_negotiation.cc:407] Connection header received
0806 22:42:17.901604 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:17.901610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:17.901709 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:17.901814 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:17.902786 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.903550 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.904789 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.905236 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.905384 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:17.909616 (+ 4232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:52807: BlockingRecv error: recv got EOF from 127.23.100.131:52807 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 22:42:17 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:17 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:17.962219 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:17.953623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:42893)
0806 22:42:17.954141 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:17.954158 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:17.954323 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:17.954711 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:17.954715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:17.955151 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:17.955311 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:17.955321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.956371 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.956376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:17.957304 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:17.957315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.958153 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.958160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:17.958324 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:17.958977 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:17.958995 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:17.961866 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":275,"threads_started":1}
W20260806 22:42:17.962374 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:17.953633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:42893 (local address 127.23.100.190:33557)
0806 22:42:17.953838 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:17.953847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:17.954249 (+ 402us) server_negotiation.cc:407] Connection header received
0806 22:42:17.954458 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:17.954463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:17.954560 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:17.954671 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:17.955463 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.956210 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.957464 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:17.957962 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:17.958105 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:17.962101 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:42893: BlockingRecv error: recv got EOF from 127.23.100.129:42893 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 22:42:18 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:18 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:18.447319 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:18.438177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:39611)
0806 22:42:18.438718 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:18.438734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:18.438851 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:18.439159 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:18.439163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:18.439588 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:18.439758 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:18.439766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.440969 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.440974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:18.442005 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:18.442020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.442817 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.442822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:18.442958 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:18.443649 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:18.443668 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:18.447015 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":298,"threads_started":1}
W20260806 22:42:18.447423 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:18.438406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:39611 (local address 127.23.100.190:33557)
0806 22:42:18.438631 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:18.438637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:18.438812 (+ 175us) server_negotiation.cc:407] Connection header received
0806 22:42:18.438955 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:18.438958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:18.439020 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:18.439097 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:18.439931 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.440807 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.442162 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.442686 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.442796 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:18.447224 (+ 4428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:39611: BlockingRecv error: recv got EOF from 127.23.100.130:39611 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 06 22:42:18 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:18 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:18.919574 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:18.910657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:45345)
0806 22:42:18.911205 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:18.911222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:18.911344 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:18.911736 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:18.911740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:18.912157 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:18.912322 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:18.912331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.913458 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.913462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:18.914708 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:18.914721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.915447 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.915451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:18.915571 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:18.916166 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:18.916184 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:18.919218 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":301,"threads_started":1}
W20260806 22:42:18.919659 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:18.910728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:45345 (local address 127.23.100.190:33557)
0806 22:42:18.910967 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:18.910977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:18.911330 (+ 353us) server_negotiation.cc:407] Connection header received
0806 22:42:18.911456 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:18.911460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:18.911585 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:18.911686 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:18.912483 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.913301 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.914843 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.915279 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.915374 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:18.919420 (+ 4046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:45345: BlockingRecv error: recv got EOF from 127.23.100.131:45345 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 22:42:18 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:18 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:18.972119 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:18.963415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:39797)
0806 22:42:18.963971 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:18.963989 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:18.964110 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:18.964427 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:18.964430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:18.964840 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:18.965036 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:18.965045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.966325 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.966330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:18.967277 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:18.967289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.968134 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.968139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:18.968308 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:18.968990 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:18.969009 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:18.971866 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":295,"threads_started":1}
W20260806 22:42:18.972327 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:18.963550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:39797 (local address 127.23.100.190:33557)
0806 22:42:18.963795 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:18.963799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:18.964066 (+ 267us) server_negotiation.cc:407] Connection header received
0806 22:42:18.964210 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:18.964213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:18.964289 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:18.964372 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:18.965201 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.966025 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.967424 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:18.967948 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:18.968064 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:18.972084 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:39797: BlockingRecv error: recv got EOF from 127.23.100.129:39797 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 06 22:42:19 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:19 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:19.463629 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:19.448461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:56305)
0806 22:42:19.449011 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:19.449030 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:19.449192 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:19.449537 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:19.449541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:19.449936 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:19.450228 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:19.450241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.451631 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.451636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:19.452736 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:19.452750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.453474 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.453478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:19.453638 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:19.459676 (+ 6038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:19.459697 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:19.463190 (+ 3493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":296,"threads_started":1}
W20260806 22:42:19.463687 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:19.448707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:56305 (local address 127.23.100.190:33557)
0806 22:42:19.448952 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:19.448960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:19.449118 (+ 158us) server_negotiation.cc:407] Connection header received
0806 22:42:19.449306 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:19.449310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:19.449374 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:19.449462 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:19.450444 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.451458 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.452898 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.453326 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.453433 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:19.463451 (+ 10018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:56305: BlockingRecv error: recv got EOF from 127.23.100.130:56305 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 06 22:42:19 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:19 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.131@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:19.930773 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:19.920578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.131:33943)
0806 22:42:19.921177 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:19.921194 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:19.921371 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:19.921786 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:19.921791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:19.922258 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:19.922462 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:19.922472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.923702 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.923708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:19.924835 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:19.924856 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.925848 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.925855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:19.926054 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:19.926792 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:19.926813 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:19.930387 (+ 3574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":318,"threads_started":1}
W20260806 22:42:19.931008 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:19.920801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:33943 (local address 127.23.100.190:33557)
0806 22:42:19.921072 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:19.921088 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:19.921291 (+ 203us) server_negotiation.cc:407] Connection header received
0806 22:42:19.921557 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:19.921561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:19.921627 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:19.921742 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:19.922629 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.923522 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.925042 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.925685 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.925829 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:19.930722 (+ 4893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.131:33943: BlockingRecv error: recv got EOF from 127.23.100.131:33943 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 06 22:42:19 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:19 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.129@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:19.983502 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:19.973705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.129:48533)
0806 22:42:19.974332 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:19.974348 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:19.974471 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:19.974921 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:19.974925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:19.975345 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:19.975555 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:19.975566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.976749 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.976753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:19.977723 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:19.977737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.978547 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.978551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:19.978679 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:19.979573 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:19.979593 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:19.983217 (+ 3624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":356,"threads_started":1}
W20260806 22:42:19.983711 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:19.973694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:48533 (local address 127.23.100.190:33557)
0806 22:42:19.973889 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:19.973894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:19.974436 (+ 542us) server_negotiation.cc:407] Connection header received
0806 22:42:19.974625 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:19.974629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:19.974715 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:19.974811 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:19.975724 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.976588 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.977873 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:19.978350 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:19.978465 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:19.983468 (+ 5003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.129:48533: BlockingRecv error: recv got EOF from 127.23.100.129:48533 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 22:42:20 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 22:42:20 dist-test-slave-zr1t krb5kdc[3596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.100.130@KRBTEST.COM for oryx/127.23.100.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 22:42:20.474342 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:20.464872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33557 (local address 127.23.100.130:43077)
0806 22:42:20.465427 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:20.465444 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:20.465585 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:20.465973 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:20.465977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:20.466468 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:20.466661 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:20.466672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:20.467903 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:20.467907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:20.468996 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:20.469011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:20.469966 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:20.469972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:20.470111 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:20.470928 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:20.470951 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:20.474086 (+ 3135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.100.190:33557: Server oryx/127.23.100.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":288,"threads_started":1}
W20260806 22:42:20.474668 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:20.464961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:43077 (local address 127.23.100.190:33557)
0806 22:42:20.465211 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:20.465219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:20.465558 (+ 339us) server_negotiation.cc:407] Connection header received
0806 22:42:20.465690 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:20.465694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:20.465760 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:20.465855 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:20.466875 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:20.467693 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:20.469153 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:20.469754 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:20.469880 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:20.474340 (+ 4460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.100.130:43077: BlockingRecv error: recv got EOF from 127.23.100.130:43077 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 22:42:20.754755 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 4559
I20260806 22:42:20.768594 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:20.755997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.23.100.129:34519)
0806 22:42:20.756616 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:20.756624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:20.756654 (+ 30us) server_negotiation.cc:407] Connection header received
0806 22:42:20.756720 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:20.756727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:20.756964 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:20.757126 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:20.758902 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:20.760014 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:20.761328 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:20.762006 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:20.762124 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:20.764816 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:20.764825 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:20.764841 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:20.764878 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:20.767975 (+ 3097us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 22:42:20.768306 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56078: Request ticket server kudu/127.23.100.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":295,"threads_started":1}
W20260806 22:42:20.768728 5314 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56078: Request ticket server kudu/127.23.100.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 22:42:20.768839 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 4559: Not authorized: Client connection negotiation failed: client connection to 127.23.100.129:34519: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 22:42:20.769163 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 4559
I20260806 22:42:20.794219 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 4809
I20260806 22:42:20.809027 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:20.795452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.23.100.130:40839)
0806 22:42:20.796083 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:20.796091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:20.796125 (+ 34us) server_negotiation.cc:407] Connection header received
0806 22:42:20.796193 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:20.796200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:20.796429 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:20.796678 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:20.798457 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:20.799678 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:20.801102 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:20.801871 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:20.802043 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:20.805047 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:20.805059 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:20.805075 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:20.805137 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:20.808105 (+ 2968us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 22:42:20.808604 (+ 499us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44762: Request ticket server kudu/127.23.100.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":294,"threads_started":1}
W20260806 22:42:20.809168 5315 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44762: Request ticket server kudu/127.23.100.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 22:42:20.809163 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 4809: Not authorized: Client connection negotiation failed: client connection to 127.23.100.130:40839: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 22:42:20.809504 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 4809
I20260806 22:42:20.835673 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 5031
I20260806 22:42:20.850085 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:20.836902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.23.100.131:45347)
0806 22:42:20.837714 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:20.837723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:20.837753 (+ 30us) server_negotiation.cc:407] Connection header received
0806 22:42:20.837839 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:20.837846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:20.838082 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:20.838252 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:20.839756 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:20.840842 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:20.842284 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:20.842939 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:20.843093 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:20.846229 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:20.846240 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:20.846256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:20.846298 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:20.849369 (+ 3071us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 22:42:20.849797 (+ 428us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34346: Request ticket server kudu/127.23.100.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":446,"threads_started":1}
W20260806 22:42:20.850281 5316 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34346: Request ticket server kudu/127.23.100.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 22:42:20.850359 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 5031: Not authorized: Client connection negotiation failed: client connection to 127.23.100.131:45347: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.100.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 22:42:20.850574 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 5031
I20260806 22:42:20.874264 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 4461
I20260806 22:42:20.875630 4549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:20.997243 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 4461
2026-08-06T22:42:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21741 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T22:42:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:42:21Z Disabled control of system clock
I20260806 22:42:21.104949 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:42295
--webserver_interface=127.23.100.190
--webserver_port=0
--builtin_ntp_servers=127.23.100.148:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:42295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 22:42:21.460063 5326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:21.460384 5326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:21.460466 5326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:21.468122 5326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:42:21.468240 5326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:21.468279 5326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:21.468348 5326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:42:21.468393 5326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:42:21.478329 5326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:42295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:42295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:21.479667 5326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:21.481470 5326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:21.494318 5333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:21.494405 5334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:21.494382 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:21.495625 5326 server_base.cc:1034] running on GCE node
I20260806 22:42:21.496557 5326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:21.498238 5326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:21.499485 5326 hybrid_clock.cc:648] HybridClock initialized: now 1786056141499427 us; error 43 us; skew 500 ppm
I20260806 22:42:21.499944 5326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:42:21.502619 5326 webserver.cc:460] Webserver started at http://127.23.100.190:39251/ using document root <none> and password file <none>
I20260806 22:42:21.503400 5326 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:21.503501 5326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:21.503885 5326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:21.507007 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "381031097fa6470b997d210531b56ae9"
format_stamp: "Formatted at 2026-08-06 22:42:21 on dist-test-slave-zr1t"
server_key: "dd5f2ea02c63ce3965653d714bbdb4b7"
server_key_iv: "a56017a39613caa68965a9a5f1d02018"
server_key_version: "encryptionkey@0"
I20260806 22:42:21.507853 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "381031097fa6470b997d210531b56ae9"
format_stamp: "Formatted at 2026-08-06 22:42:21 on dist-test-slave-zr1t"
server_key: "dd5f2ea02c63ce3965653d714bbdb4b7"
server_key_iv: "a56017a39613caa68965a9a5f1d02018"
server_key_version: "encryptionkey@0"
I20260806 22:42:21.513188 5326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:42:21.516970 5346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:21.518643 5326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 22:42:21.518855 5326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "381031097fa6470b997d210531b56ae9"
format_stamp: "Formatted at 2026-08-06 22:42:21 on dist-test-slave-zr1t"
server_key: "dd5f2ea02c63ce3965653d714bbdb4b7"
server_key_iv: "a56017a39613caa68965a9a5f1d02018"
server_key_version: "encryptionkey@0"
I20260806 22:42:21.519109 5326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:21.542367 5326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:21.543257 5326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:21.543588 5326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:21.563481 5326 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:42295
I20260806 22:42:21.563526 5424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:42295 every 8 connection(s)
I20260806 22:42:21.564888 5326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:42:21.565440 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 5326
I20260806 22:42:21.565671 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:42:21.566500 23954 external_mini_cluster.cc:1428] Setting key f775048a0649e4134f4f175b61979e9d
I20260806 22:42:21.569300 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:42:21.581496 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:21.568570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.23.100.190:42295)
0806 22:42:21.569875 (+ 1305us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:21.569890 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:21.570171 (+ 281us) server_negotiation.cc:407] Connection header received
0806 22:42:21.571955 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:21.571974 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:21.572425 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:21.572773 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:21.573624 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:21.575012 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:21.576991 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:21.577735 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:21.577894 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:21.579182 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:21.579192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:21.579205 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:21.579775 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:21.579864 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:21.579886 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:21.579977 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:21.580221 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 22:42:21.580573 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":392,"threads_started":1}
I20260806 22:42:21.582554 5425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9: Bootstrap starting.
I20260806 22:42:21.585892 5425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9: Neither blocks nor log segments found. Creating new log.
I20260806 22:42:21.586931 5425 log.cc:824] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9: Log is configured to *not* fsync() on all Append() calls
I20260806 22:42:21.590129 5425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9: No bootstrap required, opened a new log
I20260806 22:42:21.597708 5425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381031097fa6470b997d210531b56ae9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42295 } }
I20260806 22:42:21.598117 5425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:42:21.598238 5425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381031097fa6470b997d210531b56ae9, State: Initialized, Role: FOLLOWER
I20260806 22:42:21.598846 5425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381031097fa6470b997d210531b56ae9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42295 } }
I20260806 22:42:21.599108 5425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:42:21.599269 5425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:42:21.599460 5425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:42:21.601804 5425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381031097fa6470b997d210531b56ae9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42295 } }
I20260806 22:42:21.602463 5425 leader_election.cc:302] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 381031097fa6470b997d210531b56ae9; no voters:
I20260806 22:42:21.603170 5425 leader_election.cc:288] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:42:21.603343 5432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:42:21.604055 5432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [term 1 LEADER]: Becoming Leader. State: Replica: 381031097fa6470b997d210531b56ae9, State: Running, Role: LEADER
I20260806 22:42:21.604643 5432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381031097fa6470b997d210531b56ae9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42295 } }
I20260806 22:42:21.605436 5425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:42:21.607543 5434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 381031097fa6470b997d210531b56ae9. Latest consensus state: current_term: 1 leader_uuid: "381031097fa6470b997d210531b56ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381031097fa6470b997d210531b56ae9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42295 } } }
I20260806 22:42:21.608006 5434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [sys.catalog]: This master's current role is: LEADER
I20260806 22:42:21.607874 5433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "381031097fa6470b997d210531b56ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381031097fa6470b997d210531b56ae9" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42295 } } }
I20260806 22:42:21.608507 5433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9 [sys.catalog]: This master's current role is: LEADER
I20260806 22:42:21.612670 5445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:42:21.618101 5445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:42:21.627568 5445 catalog_manager.cc:1347] Generated new cluster ID: 76ac73b23752484c842edf6ef8ff9ae9
I20260806 22:42:21.627704 5445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:42:21.658486 5445 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:42:21.660485 5445 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:42:21.667923 5445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 381031097fa6470b997d210531b56ae9: Generated new TSK 0
I20260806 22:42:21.669032 5445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:42:21.684486 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:42295
--builtin_ntp_servers=127.23.100.148:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 22:42:22.032953 5462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:22.033241 5462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:22.033401 5462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:22.040685 5462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:22.040781 5462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:22.040943 5462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:42:22.050119 5462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:42295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:22.051488 5462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:22.053264 5462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:22.066344 5468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:22.066658 5469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:22.067337 5471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:22.067672 5462 server_base.cc:1034] running on GCE node
I20260806 22:42:22.068295 5462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:22.069247 5462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:22.070499 5462 hybrid_clock.cc:648] HybridClock initialized: now 1786056142070424 us; error 83 us; skew 500 ppm
I20260806 22:42:22.070853 5462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:42:22.073201 5462 webserver.cc:460] Webserver started at http://127.23.100.129:46093/ using document root <none> and password file <none>
I20260806 22:42:22.074016 5462 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:22.074159 5462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:22.074517 5462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:22.077502 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "7d14af4edab749578ac299df0bbbee03"
format_stamp: "Formatted at 2026-08-06 22:42:22 on dist-test-slave-zr1t"
server_key: "cc3f4a3b592318fc4f965f2d8bd46445"
server_key_iv: "268d85a91619e7fc431bf965b504ae22"
server_key_version: "encryptionkey@0"
I20260806 22:42:22.078382 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "7d14af4edab749578ac299df0bbbee03"
format_stamp: "Formatted at 2026-08-06 22:42:22 on dist-test-slave-zr1t"
server_key: "cc3f4a3b592318fc4f965f2d8bd46445"
server_key_iv: "268d85a91619e7fc431bf965b504ae22"
server_key_version: "encryptionkey@0"
I20260806 22:42:22.083915 5462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 22:42:22.088042 5481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:22.089711 5462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 22:42:22.089905 5462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "7d14af4edab749578ac299df0bbbee03"
format_stamp: "Formatted at 2026-08-06 22:42:22 on dist-test-slave-zr1t"
server_key: "cc3f4a3b592318fc4f965f2d8bd46445"
server_key_iv: "268d85a91619e7fc431bf965b504ae22"
server_key_version: "encryptionkey@0"
I20260806 22:42:22.090162 5462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:22.108238 5462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:22.109115 5462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:22.109438 5462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:22.110483 5462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:22.111975 5462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:22.112112 5462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:22.112241 5462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:22.112337 5462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:22.146886 5462 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:44851
I20260806 22:42:22.146958 5683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:44851 every 8 connection(s)
I20260806 22:42:22.148247 5462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:42:22.157101 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 5462
I20260806 22:42:22.157306 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:42:22.158049 23954 external_mini_cluster.cc:1428] Setting key e6156011730932d665bc7507a1fe4e6f
I20260806 22:42:22.161230 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:42295
--builtin_ntp_servers=127.23.100.148:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 22:42:22.162540 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:22.151047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:54555 (local address 127.23.100.190:42295)
0806 22:42:22.151678 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:22.151685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:22.153047 (+ 1362us) server_negotiation.cc:407] Connection header received
0806 22:42:22.154982 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:22.154987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:22.155061 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:22.155177 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:22.156296 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:22.157136 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:22.158994 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:22.159593 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:22.159730 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:22.161416 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:22.161424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:22.161432 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:22.161462 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:22.161539 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:22.161562 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:22.161663 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:22.161948 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 22:42:22.162070 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":343,"threads_started":1}
I20260806 22:42:22.163544 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:22.151850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:42295 (local address 127.23.100.129:54555)
0806 22:42:22.152862 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:22.152906 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:22.154680 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:22.155435 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:22.155444 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:22.155533 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:22.156103 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:22.156120 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:22.157341 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:22.157345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:22.158814 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:22.158835 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:22.159886 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:22.159898 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:22.160188 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:22.161032 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:42:22.161035 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:22.161079 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:42:22.161177 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:22.161689 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:22.161699 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:22.161707 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 22:42:22.161905 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 22:42:22.162275 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":329,"threads_started":1}
I20260806 22:42:22.165378 5685 heartbeater.cc:347] Connected to a master server at 127.23.100.190:42295
I20260806 22:42:22.165746 5685 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:22.166590 5685 heartbeater.cc:511] Master 127.23.100.190:42295 requested a full tablet report, sending...
I20260806 22:42:22.168866 5373 ts_manager.cc:194] Registered new tserver with Master: 7d14af4edab749578ac299df0bbbee03 (127.23.100.129:44851)
I20260806 22:42:22.170404 5373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.129:54555
W20260806 22:42:22.512787 5689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:22.513108 5689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:22.513306 5689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:22.520927 5689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:22.521044 5689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:22.521216 5689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:42:22.530978 5689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:42295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:22.532442 5689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:22.534292 5689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:22.548008 5703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:22.547922 5696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:22.547977 5698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:22.549803 5689 server_base.cc:1034] running on GCE node
I20260806 22:42:22.550505 5689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:22.551523 5689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:22.552733 5689 hybrid_clock.cc:648] HybridClock initialized: now 1786056142552651 us; error 71 us; skew 500 ppm
I20260806 22:42:22.553074 5689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:42:22.555505 5689 webserver.cc:460] Webserver started at http://127.23.100.130:41133/ using document root <none> and password file <none>
I20260806 22:42:22.556252 5689 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:22.556389 5689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:22.556730 5689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:22.559767 5689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "fa6e36691f7d40b9969ebe59d8d6c0ef"
format_stamp: "Formatted at 2026-08-06 22:42:22 on dist-test-slave-zr1t"
server_key: "62c0399a4b59674044f9b08cfd23a614"
server_key_iv: "39f8406dc5bbf3364974d62e0e7d380d"
server_key_version: "encryptionkey@0"
I20260806 22:42:22.560621 5689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "fa6e36691f7d40b9969ebe59d8d6c0ef"
format_stamp: "Formatted at 2026-08-06 22:42:22 on dist-test-slave-zr1t"
server_key: "62c0399a4b59674044f9b08cfd23a614"
server_key_iv: "39f8406dc5bbf3364974d62e0e7d380d"
server_key_version: "encryptionkey@0"
I20260806 22:42:22.566398 5689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 22:42:22.570066 5711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:22.571573 5689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 22:42:22.571750 5689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "fa6e36691f7d40b9969ebe59d8d6c0ef"
format_stamp: "Formatted at 2026-08-06 22:42:22 on dist-test-slave-zr1t"
server_key: "62c0399a4b59674044f9b08cfd23a614"
server_key_iv: "39f8406dc5bbf3364974d62e0e7d380d"
server_key_version: "encryptionkey@0"
I20260806 22:42:22.571990 5689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:22.592291 5689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:22.593245 5689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:22.593631 5689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:22.594753 5689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:22.596264 5689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:22.596390 5689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:22.596517 5689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:22.596612 5689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:22.632736 5689 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:46113
I20260806 22:42:22.632771 5889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:46113 every 8 connection(s)
I20260806 22:42:22.634112 5689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:42:22.644189 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 5689
I20260806 22:42:22.644397 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:42:22.644850 23954 external_mini_cluster.cc:1428] Setting key 48ea13b061734d6a6ed39aa6d7098c3e
I20260806 22:42:22.647212 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:22.636907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:42939 (local address 127.23.100.190:42295)
0806 22:42:22.637133 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:22.637138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:22.638525 (+ 1387us) server_negotiation.cc:407] Connection header received
0806 22:42:22.640412 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:22.640418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:22.640517 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:22.640635 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:22.641798 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:22.642549 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:22.644123 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:22.644618 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:22.644757 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:22.646341 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:22.646346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:22.646349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:22.646367 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:22.646399 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:22.646413 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:22.646492 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:22.646788 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 22:42:22.646893 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 22:42:22.647888 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:42295
--builtin_ntp_servers=127.23.100.148:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 22:42:22.648263 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:22.637315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:42295 (local address 127.23.100.130:42939)
0806 22:42:22.638336 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:22.638384 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:22.640112 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:22.640884 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:22.640893 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:22.640978 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:22.641622 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:22.641641 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:22.642720 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:22.642725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:22.643968 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:22.643981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:22.644862 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:22.644870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:22.645159 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:22.645976 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:42:22.645979 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:22.646033 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:42:22.646135 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:22.646527 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:22.646538 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:22.646546 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 22:42:22.646756 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 22:42:22.647156 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":359,"threads_started":1}
I20260806 22:42:22.649832 5890 heartbeater.cc:347] Connected to a master server at 127.23.100.190:42295
I20260806 22:42:22.650154 5890 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:22.650947 5890 heartbeater.cc:511] Master 127.23.100.190:42295 requested a full tablet report, sending...
I20260806 22:42:22.652499 5373 ts_manager.cc:194] Registered new tserver with Master: fa6e36691f7d40b9969ebe59d8d6c0ef (127.23.100.130:46113)
I20260806 22:42:22.653734 5373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.130:42939
W20260806 22:42:23.001205 5896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:23.001499 5896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:23.001653 5896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:23.009274 5896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:23.009382 5896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:23.009488 5896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:42:23.019003 5896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:42295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:23.020386 5896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:23.022301 5896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:23.036571 5902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:23.036638 5909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:23.036638 5903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:23.038170 5896 server_base.cc:1034] running on GCE node
I20260806 22:42:23.038903 5896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:23.040036 5896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:23.041308 5896 hybrid_clock.cc:648] HybridClock initialized: now 1786056143041248 us; error 81 us; skew 500 ppm
I20260806 22:42:23.041800 5896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:42:23.046769 5896 webserver.cc:460] Webserver started at http://127.23.100.131:41663/ using document root <none> and password file <none>
I20260806 22:42:23.047917 5896 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:23.048069 5896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:23.048568 5896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:23.051996 5896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "5d8a114c8cb14c0e95e4de264368d35a"
format_stamp: "Formatted at 2026-08-06 22:42:23 on dist-test-slave-zr1t"
server_key: "f5dfbbfd2e4795258364f9de27872d0e"
server_key_iv: "efab622eef1d760c78b09bec04f5b244"
server_key_version: "encryptionkey@0"
I20260806 22:42:23.052901 5896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "5d8a114c8cb14c0e95e4de264368d35a"
format_stamp: "Formatted at 2026-08-06 22:42:23 on dist-test-slave-zr1t"
server_key: "f5dfbbfd2e4795258364f9de27872d0e"
server_key_iv: "efab622eef1d760c78b09bec04f5b244"
server_key_version: "encryptionkey@0"
I20260806 22:42:23.058805 5896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 22:42:23.062541 5921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:23.063970 5896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:42:23.064184 5896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "5d8a114c8cb14c0e95e4de264368d35a"
format_stamp: "Formatted at 2026-08-06 22:42:23 on dist-test-slave-zr1t"
server_key: "f5dfbbfd2e4795258364f9de27872d0e"
server_key_iv: "efab622eef1d760c78b09bec04f5b244"
server_key_version: "encryptionkey@0"
I20260806 22:42:23.064414 5896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:23.084010 5896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:23.084908 5896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:23.085246 5896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:23.086412 5896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:23.087965 5896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:23.088114 5896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:23.088240 5896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:23.088330 5896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:23.125541 5896 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:35017
I20260806 22:42:23.125648 6113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:35017 every 8 connection(s)
I20260806 22:42:23.126935 5896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:42:23.130138 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 5896
I20260806 22:42:23.130337 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:42:23.130976 23954 external_mini_cluster.cc:1428] Setting key dff591d7046dbf0fa94ed3f40dad0724
I20260806 22:42:23.139901 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:23.129358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:60059 (local address 127.23.100.190:42295)
0806 22:42:23.129612 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:23.129619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:23.131305 (+ 1686us) server_negotiation.cc:407] Connection header received
0806 22:42:23.133588 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:23.133592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:23.133659 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:23.133741 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:23.134992 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:23.135713 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:23.137261 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:23.137725 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:23.137828 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:23.139202 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:23.139207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:23.139210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:23.139229 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:23.139259 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:23.139271 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:23.139345 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:23.139570 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 22:42:23.139652 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 22:42:23.140991 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:23.129944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:42295 (local address 127.23.100.131:60059)
0806 22:42:23.131109 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:23.131168 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:23.133238 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:23.134019 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:23.134033 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:23.134135 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:23.134830 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:23.134848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:23.135859 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:23.135863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:23.137093 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:23.137112 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:23.137941 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:23.137947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:23.138154 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:23.138893 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:42:23.138896 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:23.138936 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:42:23.139018 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:23.139370 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:23.139379 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:23.139386 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 22:42:23.139540 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 22:42:23.139864 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":372,"threads_started":1}
I20260806 22:42:23.142704 6114 heartbeater.cc:347] Connected to a master server at 127.23.100.190:42295
I20260806 22:42:23.143129 6114 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:23.144024 6114 heartbeater.cc:511] Master 127.23.100.190:42295 requested a full tablet report, sending...
I20260806 22:42:23.145781 5373 ts_manager.cc:194] Registered new tserver with Master: 5d8a114c8cb14c0e95e4de264368d35a (127.23.100.131:35017)
I20260806 22:42:23.146935 5373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.131:60059
I20260806 22:42:23.147517 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 22:42:23.155128 5687 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:23.153474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.23.100.190:42295)
0806 22:42:23.153793 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:23.153800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:23.154126 (+ 326us) server_negotiation.cc:407] Connection header received
0806 22:42:23.154272 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:23.154275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:23.154350 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:23.154448 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:23.154897 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44062: BlockingRecv error: recv got EOF from 127.0.0.1:44062 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260806 22:42:23.156083 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 5462
I20260806 22:42:23.164767 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:23.157321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.23.100.129:44851)
0806 22:42:23.157945 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:23.157952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:23.157977 (+ 25us) server_negotiation.cc:407] Connection header received
0806 22:42:23.158034 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:23.158040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:23.158242 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:23.158425 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:23.158862 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:23.159888 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:23.161228 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:23.161896 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:23.162054 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:23.163304 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:23.163312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:23.163326 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:23.163894 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:23.163952 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:23.163973 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:23.164113 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:23.164264 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 22:42:23.164413 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":282,"threads_started":1}
I20260806 22:42:23.166199 5677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:23.249293 5685 heartbeater.cc:503] Master 127.23.100.190:42295 was elected leader, sending a full tablet report...
I20260806 22:42:23.250574 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 5462
W20260806 22:42:23.271381 5349 connection.cc:535] server connection from 127.23.100.129:54555 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 22:42:23.271744 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 5689
I20260806 22:42:23.281418 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:23.272953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.23.100.130:46113)
0806 22:42:23.273658 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:23.273666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:23.273698 (+ 32us) server_negotiation.cc:407] Connection header received
0806 22:42:23.273765 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:23.273772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:23.274016 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:23.274163 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:23.274666 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:23.275922 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:23.277468 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:23.278192 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:23.278328 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:23.279577 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:23.279587 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:23.279603 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:23.280363 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:23.280426 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:23.280451 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:23.280560 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:23.280798 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 22:42:23.281018 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":323,"threads_started":1}
I20260806 22:42:23.283098 5884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:23.368479 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 5689
I20260806 22:42:23.394935 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 5896
I20260806 22:42:23.404376 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:23.396183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.23.100.131:35017)
0806 22:42:23.396863 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:23.396886 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:23.396919 (+ 33us) server_negotiation.cc:407] Connection header received
0806 22:42:23.396982 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:23.396988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:23.397207 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:23.397342 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:23.397952 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:23.399143 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:23.400600 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:23.401236 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:23.401350 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:23.402701 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:23.402710 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:23.402724 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:23.403371 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:23.403424 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:23.403446 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:23.403548 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:23.403736 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 22:42:23.403935 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":359,"threads_started":1}
I20260806 22:42:23.405995 6104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:23.489281 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 5896
I20260806 22:42:23.513401 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 5326
I20260806 22:42:23.514894 5419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:23.621345 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 5326
2026-08-06T22:42:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2611 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T22:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:42:23Z Disabled control of system clock
I20260806 22:42:23.716518 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33703
--webserver_interface=127.23.100.190
--webserver_port=0
--builtin_ntp_servers=127.23.100.148:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:33703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 22:42:24.071089 6142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:24.071412 6142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:24.071507 6142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:24.079039 6142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:42:24.079159 6142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:24.079197 6142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:24.079253 6142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:42:24.079298 6142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:42:24.088675 6142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:33703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:24.090023 6142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:24.091812 6142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:24.104461 6147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:24.104421 6151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:24.104759 6149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:24.105342 6142 server_base.cc:1034] running on GCE node
I20260806 22:42:24.106312 6142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:24.108022 6142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:24.109287 6142 hybrid_clock.cc:648] HybridClock initialized: now 1786056144109240 us; error 65 us; skew 500 ppm
I20260806 22:42:24.109754 6142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:42:24.112265 6142 webserver.cc:460] Webserver started at http://127.23.100.190:35823/ using document root <none> and password file <none>
I20260806 22:42:24.113071 6142 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:24.113219 6142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:24.113663 6142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:24.118286 6142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "3f6101e9bab540dd89405cc5fa5208cf"
format_stamp: "Formatted at 2026-08-06 22:42:24 on dist-test-slave-zr1t"
server_key: "6f68ef9508735963c7d74fdef0237309"
server_key_iv: "fcb9e302c5ce2c5d26bb357e6f09aab5"
server_key_version: "encryptionkey@0"
I20260806 22:42:24.119143 6142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "3f6101e9bab540dd89405cc5fa5208cf"
format_stamp: "Formatted at 2026-08-06 22:42:24 on dist-test-slave-zr1t"
server_key: "6f68ef9508735963c7d74fdef0237309"
server_key_iv: "fcb9e302c5ce2c5d26bb357e6f09aab5"
server_key_version: "encryptionkey@0"
I20260806 22:42:24.125029 6142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 22:42:24.128844 6160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:24.130447 6142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:42:24.130646 6142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "3f6101e9bab540dd89405cc5fa5208cf"
format_stamp: "Formatted at 2026-08-06 22:42:24 on dist-test-slave-zr1t"
server_key: "6f68ef9508735963c7d74fdef0237309"
server_key_iv: "fcb9e302c5ce2c5d26bb357e6f09aab5"
server_key_version: "encryptionkey@0"
I20260806 22:42:24.130901 6142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:24.137409 6142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:24.138331 6142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:24.138661 6142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:24.159301 6142 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:33703
I20260806 22:42:24.159320 6254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:33703 every 8 connection(s)
I20260806 22:42:24.160938 6142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:42:24.164880 6257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:42:24.168267 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 6142
I20260806 22:42:24.168509 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:42:24.169370 23954 external_mini_cluster.cc:1428] Setting key 4542c5bf22597349edfd65f4da095923
I20260806 22:42:24.179147 6257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf: Bootstrap starting.
I20260806 22:42:24.179337 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:24.171697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.23.100.190:33703)
0806 22:42:24.172808 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:24.172819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:24.172868 (+ 49us) server_negotiation.cc:407] Connection header received
0806 22:42:24.175206 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:24.175233 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:24.175725 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:24.176181 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:24.177417 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:24.177423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:24.177437 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:24.177451 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:24.177568 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:24.177594 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:24.177849 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:24.177996 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 22:42:24.178281 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":415,"threads_started":1}
I20260806 22:42:24.182602 6257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf: Neither blocks nor log segments found. Creating new log.
I20260806 22:42:24.183657 6257 log.cc:824] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf: Log is configured to *not* fsync() on all Append() calls
I20260806 22:42:24.186582 6257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf: No bootstrap required, opened a new log
I20260806 22:42:24.194352 6257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6101e9bab540dd89405cc5fa5208cf" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33703 } }
I20260806 22:42:24.194766 6257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:42:24.194842 6257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6101e9bab540dd89405cc5fa5208cf, State: Initialized, Role: FOLLOWER
I20260806 22:42:24.195341 6257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6101e9bab540dd89405cc5fa5208cf" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33703 } }
I20260806 22:42:24.195544 6257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:42:24.195640 6257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:42:24.195755 6257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:42:24.197971 6257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6101e9bab540dd89405cc5fa5208cf" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33703 } }
I20260806 22:42:24.198480 6257 leader_election.cc:302] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f6101e9bab540dd89405cc5fa5208cf; no voters:
I20260806 22:42:24.199049 6257 leader_election.cc:288] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:42:24.199216 6268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:42:24.199916 6268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [term 1 LEADER]: Becoming Leader. State: Replica: 3f6101e9bab540dd89405cc5fa5208cf, State: Running, Role: LEADER
I20260806 22:42:24.200484 6268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6101e9bab540dd89405cc5fa5208cf" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33703 } }
I20260806 22:42:24.201189 6257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:42:24.203500 6270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f6101e9bab540dd89405cc5fa5208cf. Latest consensus state: current_term: 1 leader_uuid: "3f6101e9bab540dd89405cc5fa5208cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6101e9bab540dd89405cc5fa5208cf" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33703 } } }
I20260806 22:42:24.203994 6270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [sys.catalog]: This master's current role is: LEADER
I20260806 22:42:24.203776 6269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f6101e9bab540dd89405cc5fa5208cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6101e9bab540dd89405cc5fa5208cf" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33703 } } }
I20260806 22:42:24.204113 6269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf [sys.catalog]: This master's current role is: LEADER
I20260806 22:42:24.205937 6277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:42:24.211490 6277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:42:24.221024 6277 catalog_manager.cc:1347] Generated new cluster ID: b842e0e0a0184da7b360315b9d16b458
I20260806 22:42:24.221155 6277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:42:24.272562 6277 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:42:24.273125 6277 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:42:24.286357 6277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f6101e9bab540dd89405cc5fa5208cf: Generated new TSK 0
I20260806 22:42:24.287484 6277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:42:24.294469 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33703
--builtin_ntp_servers=127.23.100.148:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 22:42:24.657061 6294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:24.657445 6294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:24.657646 6294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:24.665633 6294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:24.665787 6294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:24.665977 6294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:42:24.675596 6294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:24.677032 6294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:24.679133 6294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:24.693238 6306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:24.693204 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:24.693279 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:24.694018 6294 server_base.cc:1034] running on GCE node
I20260806 22:42:24.694693 6294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:24.695715 6294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:24.696959 6294 hybrid_clock.cc:648] HybridClock initialized: now 1786056144696897 us; error 57 us; skew 500 ppm
I20260806 22:42:24.697312 6294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:42:24.703560 6294 webserver.cc:460] Webserver started at http://127.23.100.129:35765/ using document root <none> and password file <none>
I20260806 22:42:24.704391 6294 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:24.704557 6294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:24.704900 6294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:24.708324 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "7e62123b5c8d4ee388d6b5dbda2d3315"
format_stamp: "Formatted at 2026-08-06 22:42:24 on dist-test-slave-zr1t"
server_key: "8c682015e47fae58d39895f74cc46fef"
server_key_iv: "455fe2a87a483a33170484e6dbf6bce4"
server_key_version: "encryptionkey@0"
I20260806 22:42:24.709184 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "7e62123b5c8d4ee388d6b5dbda2d3315"
format_stamp: "Formatted at 2026-08-06 22:42:24 on dist-test-slave-zr1t"
server_key: "8c682015e47fae58d39895f74cc46fef"
server_key_iv: "455fe2a87a483a33170484e6dbf6bce4"
server_key_version: "encryptionkey@0"
I20260806 22:42:24.715131 6294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:42:24.719211 6318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:24.720640 6294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:42:24.720844 6294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "7e62123b5c8d4ee388d6b5dbda2d3315"
format_stamp: "Formatted at 2026-08-06 22:42:24 on dist-test-slave-zr1t"
server_key: "8c682015e47fae58d39895f74cc46fef"
server_key_iv: "455fe2a87a483a33170484e6dbf6bce4"
server_key_version: "encryptionkey@0"
I20260806 22:42:24.721107 6294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:24.727715 6294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:24.728598 6294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:24.728915 6294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:24.730078 6294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:24.731650 6294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:24.731792 6294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:24.731954 6294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:24.732059 6294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:24.769723 6294 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:43227
I20260806 22:42:24.769901 6493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:43227 every 8 connection(s)
I20260806 22:42:24.771209 6294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:42:24.778048 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 6294
I20260806 22:42:24.778292 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:42:24.779026 23954 external_mini_cluster.cc:1428] Setting key a6420a3fce558472f9b2bfdd66ee45c5
I20260806 22:42:24.780014 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:24.774208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:55439 (local address 127.23.100.190:33703)
0806 22:42:24.774701 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:24.774706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:24.775802 (+ 1096us) server_negotiation.cc:407] Connection header received
0806 22:42:24.777584 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:24.777591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:24.777663 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:24.777790 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:24.779207 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:24.779213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:24.779217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:24.779239 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:24.779286 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:24.779299 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:24.779437 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:24.779622 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 22:42:24.779697 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":284,"threads_started":1}
I20260806 22:42:24.780978 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:24.774726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33703 (local address 127.23.100.129:55439)
0806 22:42:24.775643 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:24.775677 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:24.777262 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:24.778046 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:24.778055 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:24.778143 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:24.778904 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:42:24.778906 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:24.778945 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:42:24.779036 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:24.779396 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:24.779407 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:24.779416 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 22:42:24.779711 (+ 295us) client_negotiation.cc:240] Negotiation successful
0806 22:42:24.779974 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":286,"threads_started":1}
I20260806 22:42:24.782462 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33703
--builtin_ntp_servers=127.23.100.148:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 22:42:24.782667 6495 heartbeater.cc:347] Connected to a master server at 127.23.100.190:33703
I20260806 22:42:24.782912 6495 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:24.783627 6495 heartbeater.cc:511] Master 127.23.100.190:33703 requested a full tablet report, sending...
I20260806 22:42:24.786222 6188 ts_manager.cc:194] Registered new tserver with Master: 7e62123b5c8d4ee388d6b5dbda2d3315 (127.23.100.129:43227)
W20260806 22:42:25.129613 6502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:25.129902 6502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:25.130069 6502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:25.137727 6502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:25.137877 6502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:25.138062 6502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:42:25.147495 6502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:25.148831 6502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:25.150738 6502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:25.164247 6511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:25.164317 6512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:25.165647 6515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:25.165635 6502 server_base.cc:1034] running on GCE node
I20260806 22:42:25.166400 6502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:25.167487 6502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:25.168732 6502 hybrid_clock.cc:648] HybridClock initialized: now 1786056145168683 us; error 45 us; skew 500 ppm
I20260806 22:42:25.169088 6502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:42:25.171645 6502 webserver.cc:460] Webserver started at http://127.23.100.130:34429/ using document root <none> and password file <none>
I20260806 22:42:25.172387 6502 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:25.172531 6502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:25.172875 6502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:25.176036 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "b944974bfbcb42c4b76a210b13e56c88"
format_stamp: "Formatted at 2026-08-06 22:42:25 on dist-test-slave-zr1t"
server_key: "af529c66412ea568de46b46e5a3816b1"
server_key_iv: "fb79f9ee22e0552fa8c8966e4f47d305"
server_key_version: "encryptionkey@0"
I20260806 22:42:25.176905 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "b944974bfbcb42c4b76a210b13e56c88"
format_stamp: "Formatted at 2026-08-06 22:42:25 on dist-test-slave-zr1t"
server_key: "af529c66412ea568de46b46e5a3816b1"
server_key_iv: "fb79f9ee22e0552fa8c8966e4f47d305"
server_key_version: "encryptionkey@0"
I20260806 22:42:25.182590 6502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260806 22:42:25.186322 6523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:25.187701 6502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:42:25.187881 6502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "b944974bfbcb42c4b76a210b13e56c88"
format_stamp: "Formatted at 2026-08-06 22:42:25 on dist-test-slave-zr1t"
server_key: "af529c66412ea568de46b46e5a3816b1"
server_key_iv: "fb79f9ee22e0552fa8c8966e4f47d305"
server_key_version: "encryptionkey@0"
I20260806 22:42:25.188138 6502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:25.194299 6502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:25.195143 6502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:25.195454 6502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:25.196431 6502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:25.198125 6502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:25.198256 6502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:25.198385 6502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:25.198474 6502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:25.236235 6502 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:38215
I20260806 22:42:25.236371 6703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:38215 every 8 connection(s)
I20260806 22:42:25.237754 6502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:42:25.244650 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 6502
I20260806 22:42:25.244877 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:42:25.245375 23954 external_mini_cluster.cc:1428] Setting key 8578b64c6b048f42f46c9e4470123c9b
I20260806 22:42:25.246501 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:25.240521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:58997 (local address 127.23.100.190:33703)
0806 22:42:25.240712 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:25.240717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:25.242270 (+ 1553us) server_negotiation.cc:407] Connection header received
0806 22:42:25.243964 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:25.243969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:25.244048 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:25.244224 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:25.245721 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:25.245728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:25.245732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:25.245753 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:25.245797 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:25.245826 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:25.245969 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:25.246147 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 22:42:25.246206 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 22:42:25.247588 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:25.241052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33703 (local address 127.23.100.130:58997)
0806 22:42:25.242078 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:25.242116 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:25.243680 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:25.244399 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:25.244406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:25.244527 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:25.245333 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:42:25.245336 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:25.245375 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:42:25.245472 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:25.245934 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:25.245942 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:25.245948 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 22:42:25.246218 (+ 270us) client_negotiation.cc:240] Negotiation successful
0806 22:42:25.246506 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":292,"threads_started":1}
I20260806 22:42:25.248706 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33703
--builtin_ntp_servers=127.23.100.148:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 22:42:25.249143 6704 heartbeater.cc:347] Connected to a master server at 127.23.100.190:33703
I20260806 22:42:25.249475 6704 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:25.250082 6704 heartbeater.cc:511] Master 127.23.100.190:33703 requested a full tablet report, sending...
I20260806 22:42:25.251595 6188 ts_manager.cc:194] Registered new tserver with Master: b944974bfbcb42c4b76a210b13e56c88 (127.23.100.130:38215)
W20260806 22:42:25.595271 6711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:25.595564 6711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:25.595657 6711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:25.603199 6711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:25.603307 6711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:25.603415 6711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:42:25.612924 6711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:25.614184 6711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:25.615929 6711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:25.629065 6721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:25.629334 6717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:25.629334 6718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:25.630124 6711 server_base.cc:1034] running on GCE node
I20260806 22:42:25.630729 6711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:25.631700 6711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:25.632913 6711 hybrid_clock.cc:648] HybridClock initialized: now 1786056145632873 us; error 33 us; skew 500 ppm
I20260806 22:42:25.633244 6711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:42:25.635648 6711 webserver.cc:460] Webserver started at http://127.23.100.131:34375/ using document root <none> and password file <none>
I20260806 22:42:25.636420 6711 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:25.636565 6711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:25.636919 6711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:25.640136 6711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "36ad0bfe8ad342e98aa7174b39b76fcc"
format_stamp: "Formatted at 2026-08-06 22:42:25 on dist-test-slave-zr1t"
server_key: "2944ae6ce2c0af10ed5ceeca7a138fa7"
server_key_iv: "c7a7634c23e323e7524d921486a89689"
server_key_version: "encryptionkey@0"
I20260806 22:42:25.641016 6711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "36ad0bfe8ad342e98aa7174b39b76fcc"
format_stamp: "Formatted at 2026-08-06 22:42:25 on dist-test-slave-zr1t"
server_key: "2944ae6ce2c0af10ed5ceeca7a138fa7"
server_key_iv: "c7a7634c23e323e7524d921486a89689"
server_key_version: "encryptionkey@0"
I20260806 22:42:25.646674 6711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 22:42:25.650285 6729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:25.651655 6711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:42:25.651839 6711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "36ad0bfe8ad342e98aa7174b39b76fcc"
format_stamp: "Formatted at 2026-08-06 22:42:25 on dist-test-slave-zr1t"
server_key: "2944ae6ce2c0af10ed5ceeca7a138fa7"
server_key_iv: "c7a7634c23e323e7524d921486a89689"
server_key_version: "encryptionkey@0"
I20260806 22:42:25.652088 6711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:25.657999 6711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:25.658825 6711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:25.659127 6711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:25.660048 6711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:25.661509 6711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:25.661689 6711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:25.661832 6711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:25.661929 6711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:25.697403 6711 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:36439
I20260806 22:42:25.697461 6904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:36439 every 8 connection(s)
I20260806 22:42:25.698848 6711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:42:25.700680 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 6711
I20260806 22:42:25.700914 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:42:25.701491 23954 external_mini_cluster.cc:1428] Setting key 036e8446c8ea853ac776c4e05039a58d
I20260806 22:42:25.707698 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:25.701740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:53661 (local address 127.23.100.190:33703)
0806 22:42:25.701967 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:25.701975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:25.703757 (+ 1782us) server_negotiation.cc:407] Connection header received
0806 22:42:25.705497 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:25.705501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:25.705606 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:25.705702 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:25.707087 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:25.707092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:25.707095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:25.707109 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:25.707139 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:25.707150 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:25.707241 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:25.707422 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 22:42:25.707477 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 22:42:25.708693 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:25.702437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33703 (local address 127.23.100.131:53661)
0806 22:42:25.703598 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:25.703645 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:25.705251 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:25.705917 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:25.705925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:25.706030 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:25.706801 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:42:25.706804 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:25.706841 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:42:25.706929 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:25.707239 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:25.707248 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:25.707253 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 22:42:25.707476 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 22:42:25.707677 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":411,"threads_started":1}
I20260806 22:42:25.710263 6905 heartbeater.cc:347] Connected to a master server at 127.23.100.190:33703
I20260806 22:42:25.710578 6905 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:25.711228 6905 heartbeater.cc:511] Master 127.23.100.190:33703 requested a full tablet report, sending...
I20260806 22:42:25.712738 6188 ts_manager.cc:194] Registered new tserver with Master: 36ad0bfe8ad342e98aa7174b39b76fcc (127.23.100.131:36439)
I20260806 22:42:25.717078 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 22:42:25.723385 6500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:42:25.721862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.23.100.190:33703)
0806 22:42:25.722121 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:25.722130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:25.722551 (+ 421us) server_negotiation.cc:407] Connection header received
0806 22:42:25.722668 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:25.722672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:25.722725 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:25.722805 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:25.723150 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50044: BlockingRecv error: recv got EOF from 127.0.0.1:50044 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:42:25.723875 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 6294
I20260806 22:42:25.727576 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:25.724908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.23.100.129:43227)
0806 22:42:25.725438 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:25.725453 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:25.725480 (+ 27us) server_negotiation.cc:407] Connection header received
0806 22:42:25.725601 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:25.725608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:25.725848 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:25.726048 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:25.726965 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:25.726971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:25.726989 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:25.727002 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:25.727047 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:25.727061 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:25.727162 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:25.727236 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 22:42:25.727318 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":256,"threads_started":1}
I20260806 22:42:25.728835 6485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:25.810910 6495 heartbeater.cc:503] Master 127.23.100.190:33703 was elected leader, sending a full tablet report...
I20260806 22:42:25.812376 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 6294
W20260806 22:42:25.835114 6161 connection.cc:535] server connection from 127.23.100.129:55439 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 22:42:25.835283 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 6502
I20260806 22:42:25.839335 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:25.836461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.23.100.130:38215)
0806 22:42:25.837017 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:25.837025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:25.837054 (+ 29us) server_negotiation.cc:407] Connection header received
0806 22:42:25.837118 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:25.837124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:25.837342 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:25.837478 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:25.838587 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:25.838594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:25.838606 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:25.838627 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:25.838671 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:25.838686 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:25.838793 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:25.838919 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 22:42:25.839043 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":269,"threads_started":1}
I20260806 22:42:25.840747 6694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:25.924098 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 6502
I20260806 22:42:25.948117 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 6711
I20260806 22:42:25.952718 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:25.949427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.23.100.131:36439)
0806 22:42:25.950173 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:25.950180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:25.950212 (+ 32us) server_negotiation.cc:407] Connection header received
0806 22:42:25.950275 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:25.950281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:25.950533 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:25.950707 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:25.951925 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:25.951932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:25.951943 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:42:25.951961 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:25.951999 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:42:25.952013 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:25.952209 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:25.952287 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 22:42:25.952405 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":433,"threads_started":1}
I20260806 22:42:25.954212 6899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:26.038412 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 6711
I20260806 22:42:26.063216 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 6142
I20260806 22:42:26.064623 6247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:26.192186 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 6142
2026-08-06T22:42:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2574 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:42:26 dist-test-slave-zr1t krb5kdc[6931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:42:26 dist-test-slave-zr1t krb5kdc[6931](info): set up 2 sockets
Aug 06 22:42:26 dist-test-slave-zr1t krb5kdc[6931](info): commencing operation
krb5kdc: starting...
W20260806 22:42:27.533284 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:42:27 dist-test-slave-zr1t krb5kdc[6931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:42:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:42:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:42:27.874418 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:43673
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:43673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:42:28.233480 6948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:28.233932 6948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:28.234095 6948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:28.241616 6948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:42:28.241761 6948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:28.241901 6948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:28.242013 6948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:42:28.242103 6948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:42:28.252276 6948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:43673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:43673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:28.253688 6948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:28.255034 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:28.267345 6955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:28.267370 6958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:28.267284 6956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:28.268939 6948 server_base.cc:1034] running on GCE node
I20260806 22:42:28.269805 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:28.271234 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:28.272497 6948 hybrid_clock.cc:648] HybridClock initialized: now 1786056148272461 us; error 53 us; skew 500 ppm
I20260806 22:42:28.272866 6948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:28 dist-test-slave-zr1t krb5kdc[6931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056148, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:28.276248 6948 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:42:28.277910 6948 webserver.cc:460] Webserver started at http://127.23.100.190:35649/ using document root <none> and password file <none>
I20260806 22:42:28.278625 6948 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:28.278776 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:28.279105 6948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:28.282263 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "3cb623c8d16e40c7b99e41b0b8ca4f13"
format_stamp: "Formatted at 2026-08-06 22:42:28 on dist-test-slave-zr1t"
server_key: "84231adc831557be5fe44ba8ef0fdf6a"
server_key_iv: "9e15e275491ad6ee0231a93f741f9480"
server_key_version: "encryptionkey@0"
I20260806 22:42:28.283072 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "3cb623c8d16e40c7b99e41b0b8ca4f13"
format_stamp: "Formatted at 2026-08-06 22:42:28 on dist-test-slave-zr1t"
server_key: "84231adc831557be5fe44ba8ef0fdf6a"
server_key_iv: "9e15e275491ad6ee0231a93f741f9480"
server_key_version: "encryptionkey@0"
I20260806 22:42:28.288285 6948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:42:28.291759 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:28.293047 6948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260806 22:42:28.293232 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "3cb623c8d16e40c7b99e41b0b8ca4f13"
format_stamp: "Formatted at 2026-08-06 22:42:28 on dist-test-slave-zr1t"
server_key: "84231adc831557be5fe44ba8ef0fdf6a"
server_key_iv: "9e15e275491ad6ee0231a93f741f9480"
server_key_version: "encryptionkey@0"
I20260806 22:42:28.293431 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:28.308694 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:28.313213 6948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:28.313606 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:28.333778 6948 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:43673
I20260806 22:42:28.333834 7051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:43673 every 8 connection(s)
I20260806 22:42:28.335166 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:42:28.338790 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 6948
I20260806 22:42:28.339022 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:42:28.339257 7053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:42:28.339768 23954 external_mini_cluster.cc:1428] Setting key ae0930f6a93f7d9475ce6182c525f540
Aug 06 22:42:28 dist-test-slave-zr1t krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:42:28.355252 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13: Bootstrap starting.
I20260806 22:42:28.359762 7053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13: Neither blocks nor log segments found. Creating new log.
I20260806 22:42:28.361263 7053 log.cc:824] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13: Log is configured to *not* fsync() on all Append() calls
I20260806 22:42:28.365140 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13: No bootstrap required, opened a new log
I20260806 22:42:28.366256 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:28.341756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.23.100.190:43673)
0806 22:42:28.342758 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:28.342774 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:28.342832 (+ 58us) server_negotiation.cc:407] Connection header received
0806 22:42:28.344814 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:28.344844 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:28.345408 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:28.345916 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:28.347377 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:28.349621 (+ 2244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:28.350864 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:28.352073 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:28.352319 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:28.356334 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:28.356350 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:28.356371 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:28.356430 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:28.360913 (+ 4483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:28.361683 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:28.361692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:28.361702 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:28.361964 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:28.362511 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:28.362515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:28.362517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:28.363339 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:28.363657 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:28.364072 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 22:42:28.364638 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":411,"threads_started":1}
I20260806 22:42:28.374490 7053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb623c8d16e40c7b99e41b0b8ca4f13" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43673 } }
I20260806 22:42:28.374914 7053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:42:28.375036 7053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb623c8d16e40c7b99e41b0b8ca4f13, State: Initialized, Role: FOLLOWER
I20260806 22:42:28.375726 7053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb623c8d16e40c7b99e41b0b8ca4f13" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43673 } }
I20260806 22:42:28.376036 7053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:42:28.376194 7053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:42:28.376371 7053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:42:28.378636 7053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb623c8d16e40c7b99e41b0b8ca4f13" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43673 } }
I20260806 22:42:28.379151 7053 leader_election.cc:302] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cb623c8d16e40c7b99e41b0b8ca4f13; no voters:
I20260806 22:42:28.379738 7053 leader_election.cc:288] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:42:28.379897 7062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:42:28.381421 7062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb623c8d16e40c7b99e41b0b8ca4f13, State: Running, Role: LEADER
I20260806 22:42:28.382009 7053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:42:28.382014 7062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb623c8d16e40c7b99e41b0b8ca4f13" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43673 } }
I20260806 22:42:28.385211 7064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cb623c8d16e40c7b99e41b0b8ca4f13. Latest consensus state: current_term: 1 leader_uuid: "3cb623c8d16e40c7b99e41b0b8ca4f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb623c8d16e40c7b99e41b0b8ca4f13" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43673 } } }
I20260806 22:42:28.385650 7064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [sys.catalog]: This master's current role is: LEADER
I20260806 22:42:28.385639 7063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cb623c8d16e40c7b99e41b0b8ca4f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb623c8d16e40c7b99e41b0b8ca4f13" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43673 } } }
I20260806 22:42:28.385979 7063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13 [sys.catalog]: This master's current role is: LEADER
I20260806 22:42:28.388201 7076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:42:28.393503 7076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:42:28.403144 7076 catalog_manager.cc:1347] Generated new cluster ID: bfd9268bc18a430c9e3d7f6da44d77fd
I20260806 22:42:28.403272 7076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:42:28.417186 7076 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:42:28.419098 7076 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:42:28.429507 7076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13: Generated new TSK 0
I20260806 22:42:28.430691 7076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:42:28.508767 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43673
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:42:28.860049 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:28.860334 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:28.860430 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:28.867975 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:28.868121 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:28.868299 7092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:42:28.878520 7092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:28.879783 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:28.881055 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:28.894753 7100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:28.894652 7105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:28.894677 7101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:28.895596 7092 server_base.cc:1034] running on GCE node
I20260806 22:42:28.896206 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:28.897306 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:28.898536 7092 hybrid_clock.cc:648] HybridClock initialized: now 1786056148898481 us; error 50 us; skew 500 ppm
I20260806 22:42:28.898859 7092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:28 dist-test-slave-zr1t krb5kdc[6931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056148, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:28.902040 7092 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:42:28.903652 7092 webserver.cc:460] Webserver started at http://127.23.100.129:34855/ using document root <none> and password file <none>
I20260806 22:42:28.904357 7092 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:28.904515 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:28.904825 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:28.907858 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "9647b92ec4164996be4f1cc5b466c87c"
format_stamp: "Formatted at 2026-08-06 22:42:28 on dist-test-slave-zr1t"
server_key: "699dce419b7c1768f98ad827cac32846"
server_key_iv: "69a3067c181aa95f64baa7e3926d9a51"
server_key_version: "encryptionkey@0"
I20260806 22:42:28.908677 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "9647b92ec4164996be4f1cc5b466c87c"
format_stamp: "Formatted at 2026-08-06 22:42:28 on dist-test-slave-zr1t"
server_key: "699dce419b7c1768f98ad827cac32846"
server_key_iv: "69a3067c181aa95f64baa7e3926d9a51"
server_key_version: "encryptionkey@0"
I20260806 22:42:28.913950 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 22:42:28.917807 7117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:28.919222 7092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:42:28.919361 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "9647b92ec4164996be4f1cc5b466c87c"
format_stamp: "Formatted at 2026-08-06 22:42:28 on dist-test-slave-zr1t"
server_key: "699dce419b7c1768f98ad827cac32846"
server_key_iv: "69a3067c181aa95f64baa7e3926d9a51"
server_key_version: "encryptionkey@0"
I20260806 22:42:28.919540 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:28.934882 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:28.938746 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:28.938987 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:28.939898 7092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:28.941206 7092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:28.941280 7092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:28.941363 7092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:28.941411 7092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:28.976879 7092 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:37761
I20260806 22:42:28.976974 7295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:37761 every 8 connection(s)
I20260806 22:42:28.978227 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:42:28.980490 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 7092
I20260806 22:42:28.980744 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:42:28.981690 23954 external_mini_cluster.cc:1428] Setting key 43b7e46bb1563d42d3a0f20de0e9026c
Aug 06 22:42:28 dist-test-slave-zr1t krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056148, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:42:29.000727 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:28.980935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:46157 (local address 127.23.100.190:43673)
0806 22:42:28.981471 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:28.981478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:28.982632 (+ 1154us) server_negotiation.cc:407] Connection header received
0806 22:42:28.984504 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:28.984512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:28.984601 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:28.984714 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:28.986383 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:28.987532 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:28.989132 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:28.989854 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:28.990001 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:28.993985 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:28.993994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:28.994001 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:28.994047 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:28.997281 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:28.998384 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:28.998388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:28.998391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:28.998516 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:28.999025 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:28.999028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:28.999030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:28.999288 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:28.999397 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:29.000294 (+ 897us) server_negotiation.cc:299] Negotiation successful
0806 22:42:29.000471 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":258,"threads_started":1}
I20260806 22:42:29.002377 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:28.981469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:43673 (local address 127.23.100.129:46157)
0806 22:42:28.982460 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:28.982506 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:28.984224 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:28.984944 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:28.984952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:28.985489 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:28.986213 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:28.986231 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:28.987708 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:28.987712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:28.988977 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:28.988989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:28.990099 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:28.990104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:28.990317 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:28.991052 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:28.991078 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:28.993753 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:28.997450 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:28.997460 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:28.997479 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:28.998257 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:28.998628 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:28.998635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:28.998642 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:28.998912 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:28.999421 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:28.999432 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:28.999886 (+ 454us) client_negotiation.cc:769] Sending connection context
0806 22:42:29.000201 (+ 315us) client_negotiation.cc:240] Negotiation successful
0806 22:42:29.000848 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":438,"threads_started":1}
I20260806 22:42:29.004060 7297 heartbeater.cc:347] Connected to a master server at 127.23.100.190:43673
I20260806 22:42:29.004375 7297 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:29.005120 7297 heartbeater.cc:511] Master 127.23.100.190:43673 requested a full tablet report, sending...
I20260806 22:42:29.007220 6994 ts_manager.cc:194] Registered new tserver with Master: 9647b92ec4164996be4f1cc5b466c87c (127.23.100.129:37761)
I20260806 22:42:29.008644 6994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:46157
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:42:29.051419 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43673
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:42:29.401248 7305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:29.401609 7305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:29.401814 7305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:29.409082 7305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:29.409186 7305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:29.409368 7305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:42:29.419441 7305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:29.420759 7305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:29.422200 7305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:29.435811 7315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:29.435863 7317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:29.435755 7320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:29.436853 7305 server_base.cc:1034] running on GCE node
I20260806 22:42:29.437565 7305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:29.438706 7305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:29.439971 7305 hybrid_clock.cc:648] HybridClock initialized: now 1786056149439915 us; error 61 us; skew 500 ppm
I20260806 22:42:29.440344 7305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:29 dist-test-slave-zr1t krb5kdc[6931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056149, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:29.443794 7305 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:42:29.445354 7305 webserver.cc:460] Webserver started at http://127.23.100.130:34659/ using document root <none> and password file <none>
I20260806 22:42:29.446128 7305 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:29.446288 7305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:29.446652 7305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:29.449766 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "b4f76d7286fc45c3a225e1ef002b5120"
format_stamp: "Formatted at 2026-08-06 22:42:29 on dist-test-slave-zr1t"
server_key: "2958cb61c3d87dd1160ba5213de7975b"
server_key_iv: "f0f61f6fd9981d9539221f7ac2c9e9d8"
server_key_version: "encryptionkey@0"
I20260806 22:42:29.450583 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "b4f76d7286fc45c3a225e1ef002b5120"
format_stamp: "Formatted at 2026-08-06 22:42:29 on dist-test-slave-zr1t"
server_key: "2958cb61c3d87dd1160ba5213de7975b"
server_key_iv: "f0f61f6fd9981d9539221f7ac2c9e9d8"
server_key_version: "encryptionkey@0"
I20260806 22:42:29.455897 7305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:42:29.459640 7330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:29.461100 7305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:42:29.461309 7305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "b4f76d7286fc45c3a225e1ef002b5120"
format_stamp: "Formatted at 2026-08-06 22:42:29 on dist-test-slave-zr1t"
server_key: "2958cb61c3d87dd1160ba5213de7975b"
server_key_iv: "f0f61f6fd9981d9539221f7ac2c9e9d8"
server_key_version: "encryptionkey@0"
I20260806 22:42:29.461527 7305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:29.481606 7305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:29.485826 7305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:29.486158 7305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:29.487145 7305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:29.488476 7305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:29.488590 7305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:29.488746 7305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:29.488827 7305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:29.524343 7305 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:39525
I20260806 22:42:29.524435 7509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:39525 every 8 connection(s)
I20260806 22:42:29.525751 7305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:42:29.528366 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 7305
I20260806 22:42:29.528611 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:42:29.529153 23954 external_mini_cluster.cc:1428] Setting key 0372e14be9f257fb3c218f0b17cdbd71
Aug 06 22:42:29 dist-test-slave-zr1t krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056149, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:42:29.549175 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:29.528311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:55019 (local address 127.23.100.190:43673)
0806 22:42:29.528873 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:29.528880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:29.529943 (+ 1063us) server_negotiation.cc:407] Connection header received
0806 22:42:29.532196 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:29.532204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:29.532305 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:29.532431 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:29.533976 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:29.535193 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:29.536832 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:29.537630 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:29.537777 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:29.541479 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:29.541488 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:29.541496 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:29.541568 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:29.545316 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:29.546385 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:29.546391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:29.546394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:29.546552 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:29.547093 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:29.547097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:29.547100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:29.547449 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:29.547577 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:29.548492 (+ 915us) server_negotiation.cc:299] Negotiation successful
0806 22:42:29.548682 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":294,"threads_started":1}
I20260806 22:42:29.550590 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:29.528890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:43673 (local address 127.23.100.130:55019)
0806 22:42:29.529755 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:29.529802 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:29.531893 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:29.532634 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:29.532642 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:29.533161 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:29.533789 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:29.533809 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:29.535393 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:29.535398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:29.536671 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:29.536683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:29.537854 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:29.537860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:29.538081 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:29.538862 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:29.538904 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:29.541203 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:29.545485 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:29.545497 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:29.545537 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:29.546206 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:29.546682 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:29.546689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:29.546695 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:29.546959 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:29.547587 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:29.547598 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:29.548079 (+ 481us) client_negotiation.cc:769] Sending connection context
0806 22:42:29.548404 (+ 325us) client_negotiation.cc:240] Negotiation successful
0806 22:42:29.549082 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":346,"threads_started":1}
I20260806 22:42:29.552263 7510 heartbeater.cc:347] Connected to a master server at 127.23.100.190:43673
I20260806 22:42:29.552647 7510 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:29.553472 7510 heartbeater.cc:511] Master 127.23.100.190:43673 requested a full tablet report, sending...
I20260806 22:42:29.555117 6994 ts_manager.cc:194] Registered new tserver with Master: b4f76d7286fc45c3a225e1ef002b5120 (127.23.100.130:39525)
I20260806 22:42:29.556030 6994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:55019
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:42:29.600847 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43673
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:42:29.946784 7522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:42:29.947125 7522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:42:29.947304 7522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:42:29.954684 7522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:42:29.954842 7522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:42:29.955032 7522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:42:29.964797 7522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:42:29.966413 7522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:42:29.967706 7522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:42:29.980856 7533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:29.980830 7529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:42:29.981330 7528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:42:29.982498 7522 server_base.cc:1034] running on GCE node
I20260806 22:42:29.983078 7522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:42:29.984100 7522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:42:29.985302 7522 hybrid_clock.cc:648] HybridClock initialized: now 1786056149985247 us; error 53 us; skew 500 ppm
I20260806 22:42:29.985726 7522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:42:29 dist-test-slave-zr1t krb5kdc[6931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056149, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:42:29.993000 7522 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:42:29.994514 7522 webserver.cc:460] Webserver started at http://127.23.100.131:33939/ using document root <none> and password file <none>
I20260806 22:42:29.995232 7522 fs_manager.cc:359] Metadata directory not provided
I20260806 22:42:29.995395 7522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:42:29.995707 7522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:42:29.998878 7522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "9ffbb243bfd44607a887ae796d66fdbf"
format_stamp: "Formatted at 2026-08-06 22:42:29 on dist-test-slave-zr1t"
server_key: "3066e1614c3970d1e926b62c94cd9264"
server_key_iv: "44eeeab6bb0d447a6f3c80e58e214c55"
server_key_version: "encryptionkey@0"
I20260806 22:42:29.999843 7522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "9ffbb243bfd44607a887ae796d66fdbf"
format_stamp: "Formatted at 2026-08-06 22:42:29 on dist-test-slave-zr1t"
server_key: "3066e1614c3970d1e926b62c94cd9264"
server_key_iv: "44eeeab6bb0d447a6f3c80e58e214c55"
server_key_version: "encryptionkey@0"
I20260806 22:42:30.005259 7522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 22:42:30.008725 7543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:30.010144 7522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:42:30.010366 7522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "9ffbb243bfd44607a887ae796d66fdbf"
format_stamp: "Formatted at 2026-08-06 22:42:29 on dist-test-slave-zr1t"
server_key: "3066e1614c3970d1e926b62c94cd9264"
server_key_iv: "44eeeab6bb0d447a6f3c80e58e214c55"
server_key_version: "encryptionkey@0"
I20260806 22:42:30.010579 7522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:42:30.011668 7297 heartbeater.cc:503] Master 127.23.100.190:43673 was elected leader, sending a full tablet report...
I20260806 22:42:30.032127 7522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:42:30.036291 7522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:42:30.036612 7522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:42:30.037664 7522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:42:30.039091 7522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:42:30.039201 7522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:30.039357 7522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:42:30.039433 7522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:42:30.074491 7522 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:38883
I20260806 22:42:30.074536 7731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:38883 every 8 connection(s)
I20260806 22:42:30.075897 7522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:42:30.082885 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 7522
I20260806 22:42:30.083127 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:42:30.083595 23954 external_mini_cluster.cc:1428] Setting key 1a4ccb4b66135afbc30c9c06bee7b84e
Aug 06 22:42:30 dist-test-slave-zr1t krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056149, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:42:30.096688 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.078538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:43313 (local address 127.23.100.190:43673)
0806 22:42:30.078985 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:30.078990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:30.079915 (+ 925us) server_negotiation.cc:407] Connection header received
0806 22:42:30.081834 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:30.081839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:30.081909 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:30.082011 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:30.083671 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.084755 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.086990 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.087453 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.087554 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.090943 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:30.090948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:30.090952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:30.090992 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:30.093507 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:30.094319 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:30.094324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:30.094328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:30.094467 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:30.094927 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:30.094931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:30.094933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:30.095192 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:30.095361 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:30.096050 (+ 689us) server_negotiation.cc:299] Negotiation successful
0806 22:42:30.096273 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":240,"threads_started":1}
I20260806 22:42:30.097733 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.078976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:43673 (local address 127.23.100.131:43313)
0806 22:42:30.079704 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:30.079749 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:30.081452 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:30.082212 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:30.082219 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:30.082836 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:42:30.083483 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.083502 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.084927 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.084934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.086819 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.086835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.087653 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.087659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.087858 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.088501 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:42:30.088527 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:42:30.090720 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:42:30.093717 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:30.093727 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:30.093744 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:30.094183 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:30.094587 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:42:30.094592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:42:30.094596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:42:30.094791 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:42:30.095329 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:42:30.095338 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:42:30.095690 (+ 352us) client_negotiation.cc:769] Sending connection context
0806 22:42:30.095970 (+ 280us) client_negotiation.cc:240] Negotiation successful
0806 22:42:30.096525 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":290,"threads_started":1}
I20260806 22:42:30.099192 7733 heartbeater.cc:347] Connected to a master server at 127.23.100.190:43673
I20260806 22:42:30.099517 7733 heartbeater.cc:464] Registering TS with master...
I20260806 22:42:30.100247 7733 heartbeater.cc:511] Master 127.23.100.190:43673 requested a full tablet report, sending...
I20260806 22:42:30.101709 6994 ts_manager.cc:194] Registered new tserver with Master: 9ffbb243bfd44607a887ae796d66fdbf (127.23.100.131:38883)
I20260806 22:42:30.102489 6994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:43313
I20260806 22:42:30.111805 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:42:30.130076 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.116607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.23.100.190:43673)
0806 22:42:30.116770 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:30.116774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:30.117283 (+ 509us) server_negotiation.cc:407] Connection header received
0806 22:42:30.117514 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:30.117562 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:30.117643 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:30.117750 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:30.119204 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.119964 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.121231 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.121739 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.121884 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.124324 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:30.124329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:30.124332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:30.124364 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:30.127051 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:30.127766 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:30.127773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:30.127776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:30.127989 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:30.128521 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:30.128529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:30.128534 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:30.129050 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:30.129252 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:30.129653 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 22:42:30.129841 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 22:42:30.134236 6994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:42:30.141196 6994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:42:30.177373 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.154005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.23.100.129:37761)
0806 22:42:30.166942 (+ 12937us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:30.166956 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:30.166993 (+ 37us) server_negotiation.cc:407] Connection header received
0806 22:42:30.167117 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:30.167128 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:30.167465 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:30.167714 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:42:30.169067 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.170988 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.174273 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.175866 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.176093 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.176297 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:30.176818 (+ 521us) server_negotiation.cc:299] Negotiation successful
0806 22:42:30.177014 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12696,"thread_start_us":12536,"threads_started":1}
I20260806 22:42:30.177441 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.153879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:37761 (local address 127.0.0.1:35500)
0806 22:42:30.166327 (+ 12448us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:30.166375 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:30.166607 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:30.168115 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:30.168128 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:30.168173 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:42:30.168743 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.168768 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.171244 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.171252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.174006 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.174026 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.176109 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.176115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.176368 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.176459 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 22:42:30.176608 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 22:42:30.176928 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11372,"mutex_wait_us":123,"thread_start_us":13109,"threads_started":1}
I20260806 22:42:30.180226 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.154231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:39525 (local address 127.0.0.1:37858)
0806 22:42:30.167296 (+ 13065us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:30.167317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:30.167444 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:30.172443 (+ 4999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:30.172448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:30.172476 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:42:30.172967 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.172982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.174822 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.174831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.176496 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.176510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.179421 (+ 2911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.179428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.179609 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.179634 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 22:42:30.179744 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 22:42:30.179881 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12935,"thread_start_us":10847,"threads_started":1}
I20260806 22:42:30.180286 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.154529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.23.100.130:39525)
0806 22:42:30.171797 (+ 17268us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:30.171808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:30.171847 (+ 39us) server_negotiation.cc:407] Connection header received
0806 22:42:30.171955 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:30.171962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:30.172200 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:30.172442 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:42:30.173171 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.174666 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.176650 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.179136 (+ 2486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.179320 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.179515 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:30.179799 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 22:42:30.179964 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17058,"thread_start_us":16932,"threads_started":1}
I20260806 22:42:30.181792 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.167756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.23.100.131:38883)
0806 22:42:30.171436 (+ 3680us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:30.171447 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:30.172311 (+ 864us) server_negotiation.cc:407] Connection header received
0806 22:42:30.172905 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:30.172914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:30.173120 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:30.173320 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:42:30.174352 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.176043 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.178838 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.180257 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.180815 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.180969 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:30.181272 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 22:42:30.181429 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3495,"thread_start_us":3353,"threads_started":1}
I20260806 22:42:30.181800 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 7be28ebd4590403794a8beabaa528481 (DEFAULT_TABLE table=test-table [id=0e7d83d123b8475ea19aece132f358a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:42:30.182394 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.167754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:38883 (local address 127.0.0.1:54388)
0806 22:42:30.172155 (+ 4401us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:30.172180 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:30.172328 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:30.173625 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:30.173630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:30.173655 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:42:30.174119 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.174133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.176217 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.176223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.178675 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.178691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.180850 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.180857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.181028 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.181071 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 22:42:30.181181 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 22:42:30.182011 (+ 830us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4262,"thread_start_us":4099,"threads_started":1}
I20260806 22:42:30.183399 7190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be28ebd4590403794a8beabaa528481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:42:30.183946 7407 tablet_service.cc:1467] Processing CreateTablet for tablet 7be28ebd4590403794a8beabaa528481 (DEFAULT_TABLE table=test-table [id=0e7d83d123b8475ea19aece132f358a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:42:30.185226 7407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be28ebd4590403794a8beabaa528481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:42:30.186077 7629 tablet_service.cc:1467] Processing CreateTablet for tablet 7be28ebd4590403794a8beabaa528481 (DEFAULT_TABLE table=test-table [id=0e7d83d123b8475ea19aece132f358a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:42:30.187427 7629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be28ebd4590403794a8beabaa528481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:42:30.197058 7773 tablet_bootstrap.cc:492] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: Bootstrap starting.
I20260806 22:42:30.200484 7773 tablet_bootstrap.cc:654] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: Neither blocks nor log segments found. Creating new log.
I20260806 22:42:30.201395 7774 tablet_bootstrap.cc:492] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: Bootstrap starting.
I20260806 22:42:30.201956 7773 log.cc:824] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: Log is configured to *not* fsync() on all Append() calls
I20260806 22:42:30.203006 7779 tablet_bootstrap.cc:492] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: Bootstrap starting.
I20260806 22:42:30.205250 7774 tablet_bootstrap.cc:654] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: Neither blocks nor log segments found. Creating new log.
I20260806 22:42:30.206267 7779 tablet_bootstrap.cc:654] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: Neither blocks nor log segments found. Creating new log.
I20260806 22:42:30.206573 7773 tablet_bootstrap.cc:492] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: No bootstrap required, opened a new log
I20260806 22:42:30.206583 7774 log.cc:824] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: Log is configured to *not* fsync() on all Append() calls
I20260806 22:42:30.206854 7773 ts_tablet_manager.cc:1397] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260806 22:42:30.207481 7779 log.cc:824] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: Log is configured to *not* fsync() on all Append() calls
I20260806 22:42:30.209959 7779 tablet_bootstrap.cc:492] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: No bootstrap required, opened a new log
I20260806 22:42:30.210210 7779 ts_tablet_manager.cc:1397] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 22:42:30.216094 7774 tablet_bootstrap.cc:492] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: No bootstrap required, opened a new log
I20260806 22:42:30.216361 7774 ts_tablet_manager.cc:1397] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260806 22:42:30.218536 7779 raft_consensus.cc:348] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9647b92ec4164996be4f1cc5b466c87c" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37761 } } peers { permanent_uuid: "9ffbb243bfd44607a887ae796d66fdbf" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38883 } } peers { permanent_uuid: "b4f76d7286fc45c3a225e1ef002b5120" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39525 } }
I20260806 22:42:30.218902 7779 raft_consensus.cc:374] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:42:30.219017 7779 raft_consensus.cc:729] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffbb243bfd44607a887ae796d66fdbf, State: Initialized, Role: FOLLOWER
I20260806 22:42:30.219023 7773 raft_consensus.cc:348] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9647b92ec4164996be4f1cc5b466c87c" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37761 } } peers { permanent_uuid: "9ffbb243bfd44607a887ae796d66fdbf" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38883 } } peers { permanent_uuid: "b4f76d7286fc45c3a225e1ef002b5120" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39525 } }
I20260806 22:42:30.219465 7773 raft_consensus.cc:374] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:42:30.219556 7773 raft_consensus.cc:729] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f76d7286fc45c3a225e1ef002b5120, State: Initialized, Role: FOLLOWER
I20260806 22:42:30.219645 7779 consensus_queue.cc:260] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9647b92ec4164996be4f1cc5b466c87c" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37761 } } peers { permanent_uuid: "9ffbb243bfd44607a887ae796d66fdbf" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38883 } } peers { permanent_uuid: "b4f76d7286fc45c3a225e1ef002b5120" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39525 } }
I20260806 22:42:30.220100 7773 consensus_queue.cc:260] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9647b92ec4164996be4f1cc5b466c87c" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37761 } } peers { permanent_uuid: "9ffbb243bfd44607a887ae796d66fdbf" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38883 } } peers { permanent_uuid: "b4f76d7286fc45c3a225e1ef002b5120" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39525 } }
I20260806 22:42:30.220852 7733 heartbeater.cc:503] Master 127.23.100.190:43673 was elected leader, sending a full tablet report...
I20260806 22:42:30.222190 7510 heartbeater.cc:503] Master 127.23.100.190:43673 was elected leader, sending a full tablet report...
I20260806 22:42:30.222893 7779 ts_tablet_manager.cc:1428] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260806 22:42:30.222910 7773 ts_tablet_manager.cc:1428] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260806 22:42:30.227774 7774 raft_consensus.cc:348] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9647b92ec4164996be4f1cc5b466c87c" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37761 } } peers { permanent_uuid: "9ffbb243bfd44607a887ae796d66fdbf" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38883 } } peers { permanent_uuid: "b4f76d7286fc45c3a225e1ef002b5120" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39525 } }
I20260806 22:42:30.228223 7774 raft_consensus.cc:374] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:42:30.228353 7774 raft_consensus.cc:729] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9647b92ec4164996be4f1cc5b466c87c, State: Initialized, Role: FOLLOWER
I20260806 22:42:30.228876 7774 consensus_queue.cc:260] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9647b92ec4164996be4f1cc5b466c87c" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37761 } } peers { permanent_uuid: "9ffbb243bfd44607a887ae796d66fdbf" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38883 } } peers { permanent_uuid: "b4f76d7286fc45c3a225e1ef002b5120" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39525 } }
W20260806 22:42:30.230450 7298 tablet.cc:2369] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:42:30.231405 7774 ts_tablet_manager.cc:1428] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260806 22:42:30.277860 7511 tablet.cc:2369] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:42:30.327564 7734 tablet.cc:2369] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:42:30.343482 7786 raft_consensus.cc:482] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:42:30.343799 7786 raft_consensus.cc:504] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9647b92ec4164996be4f1cc5b466c87c" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37761 } } peers { permanent_uuid: "9ffbb243bfd44607a887ae796d66fdbf" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38883 } } peers { permanent_uuid: "b4f76d7286fc45c3a225e1ef002b5120" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39525 } }
I20260806 22:42:30.345857 7786 leader_election.cc:288] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ffbb243bfd44607a887ae796d66fdbf (127.23.100.131:38883), b4f76d7286fc45c3a225e1ef002b5120 (127.23.100.130:39525)
I20260806 22:42:30.353005 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.346415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:36311 (local address 127.23.100.130:39525)
0806 22:42:30.346609 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:30.346615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:30.347551 (+ 936us) server_negotiation.cc:407] Connection header received
0806 22:42:30.347788 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:30.347792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:30.347853 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:30.347951 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:42:30.348547 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.349277 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.351310 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.352059 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.352226 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.352304 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:30.352594 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 22:42:30.352720 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 22:42:30.353375 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.346803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:39621 (local address 127.23.100.131:38883)
0806 22:42:30.346989 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:30.346998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:30.347981 (+ 983us) server_negotiation.cc:407] Connection header received
0806 22:42:30.348191 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:30.348202 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:30.348262 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:30.348371 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:42:30.348940 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.349757 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.351662 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.352425 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.352614 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.352691 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:30.352996 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 22:42:30.353125 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 22:42:30.353616 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.346185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:39525 (local address 127.23.100.129:36311)
0806 22:42:30.347395 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:30.347418 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:30.347611 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:30.347979 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:30.347983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:30.348041 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:42:30.348371 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.348380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.349402 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.349410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.351134 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.351146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.352270 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.352276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.352428 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.352454 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 22:42:30.352564 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 22:42:30.352704 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":1,"spinlock_wait_cycles":22272,"thread_start_us":1355,"threads_started":1}
I20260806 22:42:30.353643 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.346268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:38883 (local address 127.23.100.129:39621)
0806 22:42:30.347851 (+ 1583us) negotiation.cc:107] Waiting for socket to connect
0806 22:42:30.347866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 22:42:30.348045 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:42:30.348416 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:42:30.348422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:42:30.348448 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:42:30.348767 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.348776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.349946 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.349952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.351492 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:42:30.351505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.352658 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.352664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:42:30.352821 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.352849 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 22:42:30.352968 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 22:42:30.353097 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1490,"spinlock_wait_cycles":896,"thread_start_us":752,"threads_started":1}
I20260806 22:42:30.354833 7656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be28ebd4590403794a8beabaa528481" candidate_uuid: "9647b92ec4164996be4f1cc5b466c87c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffbb243bfd44607a887ae796d66fdbf" is_pre_election: true
I20260806 22:42:30.354815 7437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be28ebd4590403794a8beabaa528481" candidate_uuid: "9647b92ec4164996be4f1cc5b466c87c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f76d7286fc45c3a225e1ef002b5120" is_pre_election: true
I20260806 22:42:30.355420 7656 raft_consensus.cc:2454] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9647b92ec4164996be4f1cc5b466c87c in term 0.
I20260806 22:42:30.355419 7437 raft_consensus.cc:2454] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9647b92ec4164996be4f1cc5b466c87c in term 0.
I20260806 22:42:30.356155 7120 leader_election.cc:302] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9647b92ec4164996be4f1cc5b466c87c, 9ffbb243bfd44607a887ae796d66fdbf; no voters:
I20260806 22:42:30.356626 7786 raft_consensus.cc:2781] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:42:30.356794 7786 raft_consensus.cc:482] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:42:30.356915 7786 raft_consensus.cc:3037] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:42:30.359469 7786 raft_consensus.cc:504] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9647b92ec4164996be4f1cc5b466c87c" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37761 } } peers { permanent_uuid: "9ffbb243bfd44607a887ae796d66fdbf" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38883 } } peers { permanent_uuid: "b4f76d7286fc45c3a225e1ef002b5120" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39525 } }
I20260806 22:42:30.360131 7786 leader_election.cc:288] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [CANDIDATE]: Term 1 election: Requested vote from peers 9ffbb243bfd44607a887ae796d66fdbf (127.23.100.131:38883), b4f76d7286fc45c3a225e1ef002b5120 (127.23.100.130:39525)
I20260806 22:42:30.360666 7437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be28ebd4590403794a8beabaa528481" candidate_uuid: "9647b92ec4164996be4f1cc5b466c87c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4f76d7286fc45c3a225e1ef002b5120"
I20260806 22:42:30.360667 7656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be28ebd4590403794a8beabaa528481" candidate_uuid: "9647b92ec4164996be4f1cc5b466c87c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffbb243bfd44607a887ae796d66fdbf"
I20260806 22:42:30.360890 7656 raft_consensus.cc:3037] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:42:30.360890 7437 raft_consensus.cc:3037] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:42:30.364223 7656 raft_consensus.cc:2454] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9647b92ec4164996be4f1cc5b466c87c in term 1.
I20260806 22:42:30.364228 7437 raft_consensus.cc:2454] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9647b92ec4164996be4f1cc5b466c87c in term 1.
I20260806 22:42:30.364674 7121 leader_election.cc:302] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9647b92ec4164996be4f1cc5b466c87c, b4f76d7286fc45c3a225e1ef002b5120; no voters:
I20260806 22:42:30.364964 7786 raft_consensus.cc:2781] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:42:30.365547 7786 raft_consensus.cc:686] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 1 LEADER]: Becoming Leader. State: Replica: 9647b92ec4164996be4f1cc5b466c87c, State: Running, Role: LEADER
I20260806 22:42:30.366107 7786 consensus_queue.cc:237] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9647b92ec4164996be4f1cc5b466c87c" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37761 } } peers { permanent_uuid: "9ffbb243bfd44607a887ae796d66fdbf" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38883 } } peers { permanent_uuid: "b4f76d7286fc45c3a225e1ef002b5120" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39525 } }
I20260806 22:42:30.370708 6994 catalog_manager.cc:5495] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9647b92ec4164996be4f1cc5b466c87c (127.23.100.129). New cstate: current_term: 1 leader_uuid: "9647b92ec4164996be4f1cc5b466c87c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9647b92ec4164996be4f1cc5b466c87c" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 37761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ffbb243bfd44607a887ae796d66fdbf" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4f76d7286fc45c3a225e1ef002b5120" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39525 } health_report { overall_health: UNKNOWN } } }
I20260806 22:42:30.418349 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.412633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.23.100.129:37761)
0806 22:42:30.412900 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:30.412905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:30.412925 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:42:30.413139 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:30.413143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:30.413220 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:30.413346 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:42:30.413750 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.414598 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.416030 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.416461 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.416603 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.417129 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:42:30.417733 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:42:30.417894 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:30.417995 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 22:42:30.418079 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 22:42:30.426395 7656 raft_consensus.cc:1261] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf [term 1 FOLLOWER]: Refusing update from remote peer 9647b92ec4164996be4f1cc5b466c87c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:42:30.426397 7437 raft_consensus.cc:1261] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120 [term 1 FOLLOWER]: Refusing update from remote peer 9647b92ec4164996be4f1cc5b466c87c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:42:30.427373 7790 consensus_queue.cc:1035] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffbb243bfd44607a887ae796d66fdbf" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:42:30.427743 7786 consensus_queue.cc:1035] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f76d7286fc45c3a225e1ef002b5120" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 39525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:42:30.452715 6994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:42:30.452980 6994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:42:30.456739 6994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cb623c8d16e40c7b99e41b0b8ca4f13: Sending DeleteTablet for 3 replicas of tablet 7be28ebd4590403794a8beabaa528481
I20260806 22:42:30.457998 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be28ebd4590403794a8beabaa528481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:42:30 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:35500
I20260806 22:42:30.458086 7629 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be28ebd4590403794a8beabaa528481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:42:30 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:54388
I20260806 22:42:30.458810 7407 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be28ebd4590403794a8beabaa528481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:42:30 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:37858
I20260806 22:42:30.459838 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 7092
I20260806 22:42:30.470379 7813 tablet_replica.cc:330] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: stopping tablet replica
I20260806 22:42:30.470865 7813 raft_consensus.cc:2229] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:42:30.471402 7813 raft_consensus.cc:2258] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:42:30.473764 7811 tablet_replica.cc:330] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: stopping tablet replica
I20260806 22:42:30.474222 7811 raft_consensus.cc:2229] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:42:30.474925 7811 raft_consensus.cc:2258] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:42:30.476317 7809 tablet_replica.cc:330] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: stopping tablet replica
I20260806 22:42:30.476745 7809 raft_consensus.cc:2229] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:42:30.477240 7809 raft_consensus.cc:2258] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:42:30.477981 7813 ts_tablet_manager.cc:1905] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:42:30.478920 7811 ts_tablet_manager.cc:1905] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:42:30.485875 7813 ts_tablet_manager.cc:1918] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:42:30.486059 7813 log.cc:1196] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/7be28ebd4590403794a8beabaa528481
I20260806 22:42:30.486745 7813 ts_tablet_manager.cc:1939] T 7be28ebd4590403794a8beabaa528481 P b4f76d7286fc45c3a225e1ef002b5120: Deleting consensus metadata
Aug 06 22:42:30 dist-test-slave-zr1t krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:42:30.487669 7811 ts_tablet_manager.cc:1918] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:42:30.487849 7811 log.cc:1196] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/7be28ebd4590403794a8beabaa528481
I20260806 22:42:30.488412 7811 ts_tablet_manager.cc:1939] T 7be28ebd4590403794a8beabaa528481 P 9647b92ec4164996be4f1cc5b466c87c: Deleting consensus metadata
I20260806 22:42:30.488807 6975 catalog_manager.cc:4841] TS b4f76d7286fc45c3a225e1ef002b5120 (127.23.100.130:39525): tablet 7be28ebd4590403794a8beabaa528481 (table test-table [id=0e7d83d123b8475ea19aece132f358a6]) successfully deleted
I20260806 22:42:30.490191 6974 catalog_manager.cc:4841] TS 9647b92ec4164996be4f1cc5b466c87c (127.23.100.129:37761): tablet 7be28ebd4590403794a8beabaa528481 (table test-table [id=0e7d83d123b8475ea19aece132f358a6]) successfully deleted
I20260806 22:42:30.490701 7809 ts_tablet_manager.cc:1905] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:42:30.496994 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.461341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.23.100.129:37761)
0806 22:42:30.461622 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:30.461631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:30.475759 (+ 14128us) server_negotiation.cc:407] Connection header received
0806 22:42:30.476133 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:30.476138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:30.476244 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:30.476326 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:30.478599 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.479823 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.481839 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.482553 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.482657 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.488750 (+ 6093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:30.488769 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:30.488793 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:30.488857 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:30.493239 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:30.494215 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:30.494226 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:30.494229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:30.494417 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:30.494946 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:30.494953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:30.494957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:30.495420 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:30.495712 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:30.496328 (+ 616us) server_negotiation.cc:299] Negotiation successful
0806 22:42:30.496591 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 22:42:30.497925 7809 ts_tablet_manager.cc:1918] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:42:30.498111 7809 log.cc:1196] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/7be28ebd4590403794a8beabaa528481
I20260806 22:42:30.498626 7287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:30.498667 7809 ts_tablet_manager.cc:1939] T 7be28ebd4590403794a8beabaa528481 P 9ffbb243bfd44607a887ae796d66fdbf: Deleting consensus metadata
I20260806 22:42:30.500160 6974 catalog_manager.cc:4841] TS 9ffbb243bfd44607a887ae796d66fdbf (127.23.100.131:38883): tablet 7be28ebd4590403794a8beabaa528481 (table test-table [id=0e7d83d123b8475ea19aece132f358a6]) successfully deleted
I20260806 22:42:30.600607 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 7092
I20260806 22:42:30.626745 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 7305
Aug 06 22:42:30 dist-test-slave-zr1t krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:42:30.642498 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.627626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53904 (local address 127.23.100.130:39525)
0806 22:42:30.627798 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:30.627803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:30.627821 (+ 18us) server_negotiation.cc:407] Connection header received
0806 22:42:30.627944 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:30.627949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:30.628012 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:30.628095 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:30.629392 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.630222 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.631568 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.631934 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.632061 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.636266 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:30.636283 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:30.636301 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:30.636343 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:30.639398 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:30.640337 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:30.640345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:30.640348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:30.640487 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:30.640848 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:30.640852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:30.640855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:30.641319 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:30.641661 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:30.641904 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 22:42:30.642075 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 22:42:30.643246 7502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:30.733654 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 7305
I20260806 22:42:30.759891 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 7522
Aug 06 22:42:30 dist-test-slave-zr1t krb5kdc[6931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:42:30.775532 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:42:30.760739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.23.100.131:38883)
0806 22:42:30.760913 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 22:42:30.760918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:42:30.760940 (+ 22us) server_negotiation.cc:407] Connection header received
0806 22:42:30.761113 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:42:30.761117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:42:30.761195 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:42:30.761282 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:42:30.762693 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.763454 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.764720 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:42:30.765086 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:42:30.765177 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:42:30.769854 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:42:30.769871 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:42:30.769887 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:42:30.769922 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:42:30.772775 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:30.773487 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:30.773494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:30.773496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:30.773913 (+ 417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:42:30.774256 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:42:30.774259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:42:30.774261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:42:30.774616 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:42:30.774840 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 22:42:30.775095 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 22:42:30.775266 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 22:42:30.776227 7720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:30.865983 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 7522
I20260806 22:42:30.892601 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 6948
I20260806 22:42:30.893908 7043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:42:31.007844 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 6948
2026-08-06T22:42:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4818 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:42:31 dist-test-slave-zr1t krb5kdc[7826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:42:31 dist-test-slave-zr1t krb5kdc[7826](info): set up 2 sockets
Aug 06 22:42:31 dist-test-slave-zr1t krb5kdc[7826](info): commencing operation
krb5kdc: starting...
W20260806 22:42:32.346529 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.003s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:42:32 dist-test-slave-zr1t krb5kdc[7826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:42:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.100.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.100.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/rangeradmin_127.23.100.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/rangeradmin_127.23.100.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.100.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.100.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/rangerlookup_127.23.100.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/rangerlookup_127.23.100.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.100.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.100.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/HTTP_127.23.100.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/HTTP_127.23.100.148@KRBTEST.COM.keytab.
I20260806 22:42:32.790447 23954 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9rdcRI/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/postgres -l logfile start
2026-08-06 22:42:35.413 UTC [7870] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 22:42:35.414 UTC [7870] LOG: listening on IPv4 address "127.23.100.148", port 44799
2026-08-06 22:42:35.417 UTC [7870] LOG: listening on Unix socket "/tmp/.s.PGSQL.44799"
2026-08-06 22:42:35.446 UTC [7875] LOG: database system was shut down at 2026-08-06 22:42:33 UTC
2026-08-06 22:42:35.450 UTC [7870] LOG: database system is ready to accept connections
I20260806 22:42:36.644979 23954 mini_postgres.cc:96] Postgres bound to 44799
2026-08-06 22:42:36.672 UTC [7888] FATAL: database "slave" does not exist
127.23.100.148:44799 - accepting connections
I20260806 22:42:36.674077 23954 mini_ranger.cc:161] Starting Ranger...
I20260806 22:42:36.710282 23954 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 22:42:37.184226 23954 mini_ranger.cc:87] Created ranger Postgres database
I20260806 22:42:37.184379 23954 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-admin
2026-08-06 22:42:37,529 [I] DB FLAVOR :POSTGRES
2026-08-06 22:42:37,530 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 22:42:37,530 [I] Checking connection..
2026-08-06 22:42:37,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 22:42:37,767 [I] Checking connection passed.
2026-08-06 22:42:37,768 [I] --------- Verifying version history table ---------
2026-08-06 22:42:37,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 22:42:38,010 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 22:42:38,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 22:42:38,248 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 22:42:38,249 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 22:42:38,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 22:42:38.484 UTC [7992] WARNING: there is no transaction in progress
2026-08-06 22:42:38,492 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 22:42:38,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 22:42:38,762 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 22:42:38,763 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 22:42:38,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 22:42:39,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 22:42:39,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zr1t.c.gcp-upstream.internal','N') ;"
2026-08-06 22:42:39,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 22:42:39,765 [I] Table x_portal_user does not exist in database ranger
2026-08-06 22:42:39,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 22:42:40,022 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 22:42:40,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 22:42:40,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 22:42:40,535 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 22:42:40,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 22:42:41.450 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.466 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.484 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.499 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.514 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.529 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.591 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.597 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.612 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.621 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.631 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.644 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.653 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.659 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.666 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:41.672 UTC [8166] WARNING: there is no transaction in progress
2026-08-06 22:42:42,274 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 22:42:42,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 22:42:42,545 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 22:42:42,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 22:42:42,792 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 22:42:42,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 22:42:43,059 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 22:42:43,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 22:42:43,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 22:42:43,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zr1t.c.gcp-upstream.internal';"
2026-08-06 22:42:43,831 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 22:42:43,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 22:42:44,086 [I] DB_PATCHES have already been applied
I20260806 22:42:44.093129 23954 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-admin:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9rdcRI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 22:42:44.093277 23954 mini_ranger.cc:193] Using host: 127.23.100.148
I20260806 22:42:44.118147 23954 mini_ranger.cc:227] Ranger admin URL: http://127.23.100.148:46375
Aug 06, 2026 10:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 10:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 10:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 10:42:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 10:42:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 22:42:45.583086 23954 mini_ranger.cc:160] Time spent starting Ranger: real 8.909s user 0.012s sys 0.065s
Aug 06, 2026 10:43:06 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 10:43:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 10:43:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 10:43:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 10:43:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 22:43:08.378280 23954 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.100.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.100.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/rangerkms_127.23.100.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/rangerkms_127.23.100.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.100.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.100.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.100.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/HTTP_127.23.100.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.100.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/HTTP_127.23.100.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 22:43:08 dist-test-slave-zr1t krb5kdc[7826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056188, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 22:43:08.590788 23954 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 22:43:08.618847 23954 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 22:43:09.450999 23954 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 22:43:09.451186 23954 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-kms
I20260806 22:43:09.451315 23954 mini_ranger_kms.cc:227] Using postgres at 127.23.100.148:44799
2026-08-06 22:43:09,969 [I] DB FLAVOR :POSTGRES
2026-08-06 22:43:09,970 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 22:43:09,970 [I] Checking connection
2026-08-06 22:43:09,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 22:43:10,219 [I] connection success
2026-08-06 22:43:10,220 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 22:43:10,220 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 22:43:10,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 22:43:10,480 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 22:43:10,481 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 22:43:10,481 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 22:43:10,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:44799/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 22:43:10,740 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 22:43:11.257731 23954 mini_ranger_kms.cc:310] Created kms service
I20260806 22:43:11.475252 23954 mini_ranger_kms.cc:326] Created kudu user
I20260806 22:43:11.524184 23954 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 22:43:11 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056188, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:43:11.979843 23954 mini_ranger_kms.cc:384] Added ranger policy
I20260806 22:43:11.980955 23954 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-kms:/tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 22:43:11.981042 23954 mini_ranger_kms.cc:240] Using host: 127.23.100.148
I20260806 22:43:12.009461 23954 mini_ranger_kms.cc:276] Ranger KMS PID: 8424
I20260806 22:43:12.009703 23954 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.100.148:52773
Aug 06, 2026 10:43:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:43:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:43:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 10:43:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 10:43:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 10:43:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 22:43:14 dist-test-slave-zr1t krb5kdc[7826](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786056194, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 22:43:14 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056194, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.148@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:43:14.779013 23954 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.188s user 0.163s sys 0.399s
I20260806 22:43:14.779404 23954 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 22:43:14.779645 23954 mini_ranger_kms.cc:401] 127.23.100.148:52773/kms/v1/keys
Aug 06 22:43:15 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056194, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.148@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
Aug 06 22:43:17 dist-test-slave-zr1t krb5kdc[7826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:43:17.485734 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:39035
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:39035
--ranger_config_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:52773/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:43:17.840994 8495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:43:17.841355 8495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:43:17.841558 8495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:43:17.849038 8495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:43:17.849213 8495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:43:17.849328 8495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:43:17.849445 8495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:43:17.849557 8495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:43:17.860360 8495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.100.148:52773/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:39035
--ranger_config_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:39035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:43:17.861825 8495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:43:17.863166 8495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:43:17.876286 8504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:43:17.876319 8507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:43:17.876345 8502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:43:17.876757 8495 server_base.cc:1034] running on GCE node
I20260806 22:43:17.877863 8495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:43:17.879315 8495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:43:17.880540 8495 hybrid_clock.cc:648] HybridClock initialized: now 1786056197880496 us; error 29 us; skew 500 ppm
I20260806 22:43:17.880852 8495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:43:17 dist-test-slave-zr1t krb5kdc[7826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056197, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:43:17.884268 8495 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:43:17.885851 8495 webserver.cc:460] Webserver started at http://127.23.100.190:44615/ using document root <none> and password file <none>
I20260806 22:43:17.886521 8495 fs_manager.cc:359] Metadata directory not provided
I20260806 22:43:17.886624 8495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:43:17.886893 8495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 22:43:17 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056197, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:43:17.948024 8495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "88f8ad1fb7c2482aa684aa8ec806c687"
format_stamp: "Formatted at 2026-08-06 22:43:17 on dist-test-slave-zr1t"
server_key: "ced45a1c0ee564264565136a0952e13e"
server_key_iv: "0949b15ffad7938d609421983f525445"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:17.948864 8495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "88f8ad1fb7c2482aa684aa8ec806c687"
format_stamp: "Formatted at 2026-08-06 22:43:17 on dist-test-slave-zr1t"
server_key: "ced45a1c0ee564264565136a0952e13e"
server_key_iv: "0949b15ffad7938d609421983f525445"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:17.954635 8495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 22:43:17.976346 8519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:43:17.978036 8495 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 22:43:17.978245 8495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "88f8ad1fb7c2482aa684aa8ec806c687"
format_stamp: "Formatted at 2026-08-06 22:43:17 on dist-test-slave-zr1t"
server_key: "ced45a1c0ee564264565136a0952e13e"
server_key_iv: "0949b15ffad7938d609421983f525445"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:17.978477 8495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:43:18.016606 8495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:43:18.021023 8495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:43:18.021399 8495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:43:18.041885 8495 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:39035
I20260806 22:43:18.041935 8594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:39035 every 8 connection(s)
I20260806 22:43:18.043483 8495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:43:18.045226 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 8495
I20260806 22:43:18.045442 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:43:18.046346 23954 external_mini_cluster.cc:1428] Setting key e4fe703624cf4e0c6f4f39402378cb14
I20260806 22:43:18.048360 8595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 22:43:18 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:43:18.062783 8595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687: Bootstrap starting.
I20260806 22:43:18.066051 8595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687: Neither blocks nor log segments found. Creating new log.
I20260806 22:43:18.067128 8595 log.cc:824] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687: Log is configured to *not* fsync() on all Append() calls
I20260806 22:43:18.070888 8595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687: No bootstrap required, opened a new log
I20260806 22:43:18.071111 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:18.048634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.23.100.190:39035)
0806 22:43:18.049465 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:18.049476 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:18.049565 (+ 89us) server_negotiation.cc:407] Connection header received
0806 22:43:18.051183 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:18.051203 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:18.051586 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:18.051948 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:43:18.053677 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:18.055135 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:18.056930 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:18.057666 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:18.057798 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:18.062810 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:43:18.062819 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:43:18.062832 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:43:18.062910 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:43:18.066091 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:18.066867 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:18.066876 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:18.066881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:18.067046 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:18.067530 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:18.067538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:18.067542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:18.068382 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:43:18.068712 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:18.069195 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 22:43:18.069798 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":425,"threads_started":1}
I20260806 22:43:18.078621 8595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8ad1fb7c2482aa684aa8ec806c687" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39035 } }
I20260806 22:43:18.078991 8595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:43:18.079071 8595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f8ad1fb7c2482aa684aa8ec806c687, State: Initialized, Role: FOLLOWER
I20260806 22:43:18.079561 8595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8ad1fb7c2482aa684aa8ec806c687" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39035 } }
I20260806 22:43:18.079756 8595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:43:18.079864 8595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:43:18.079990 8595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:43:18.082289 8595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8ad1fb7c2482aa684aa8ec806c687" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39035 } }
I20260806 22:43:18.082862 8595 leader_election.cc:302] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88f8ad1fb7c2482aa684aa8ec806c687; no voters:
I20260806 22:43:18.083540 8595 leader_election.cc:288] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:43:18.083675 8601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:43:18.085013 8601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [term 1 LEADER]: Becoming Leader. State: Replica: 88f8ad1fb7c2482aa684aa8ec806c687, State: Running, Role: LEADER
I20260806 22:43:18.085580 8601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8ad1fb7c2482aa684aa8ec806c687" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39035 } }
I20260806 22:43:18.085868 8595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:43:18.088341 8603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f8ad1fb7c2482aa684aa8ec806c687. Latest consensus state: current_term: 1 leader_uuid: "88f8ad1fb7c2482aa684aa8ec806c687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8ad1fb7c2482aa684aa8ec806c687" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39035 } } }
I20260806 22:43:18.088781 8602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88f8ad1fb7c2482aa684aa8ec806c687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8ad1fb7c2482aa684aa8ec806c687" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 39035 } } }
I20260806 22:43:18.089144 8603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [sys.catalog]: This master's current role is: LEADER
I20260806 22:43:18.089185 8602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687 [sys.catalog]: This master's current role is: LEADER
I20260806 22:43:18.095834 8611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:43:18.100119 8611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:43:18.101091 8595 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 22:43:18.119712 8611 catalog_manager.cc:1347] Generated new cluster ID: 747ae136a4b5445082620281778afee2
I20260806 22:43:18.119879 8611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:43:18.163592 8611 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:43:18.165745 8611 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:43:18.184582 8611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687: Generated new TSK 0
I20260806 22:43:18.185887 8611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:43:18.955919 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:52773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39035
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:43:19 dist-test-slave-zr1t krb5kdc[7826](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786056199, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 22:43:19.418421 8661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:43:19.418926 8661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:43:19.419144 8661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:43:19.429392 8661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:43:19.429626 8661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:43:19.429898 8661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:43:19.440104 8661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.100.148:52773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:43:19.441705 8661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:43:19.443015 8661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:43:19.460655 8675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:43:19.460705 8678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:43:19.466495 8661 server_base.cc:1034] running on GCE node
W20260806 22:43:19.466792 8681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:43:19.467662 8661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:43:19.469058 8661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:43:19.470384 8661 hybrid_clock.cc:648] HybridClock initialized: now 1786056199470270 us; error 100 us; skew 500 ppm
I20260806 22:43:19.470891 8661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:43:19 dist-test-slave-zr1t krb5kdc[7826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056199, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:43:19.475591 8661 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:43:19.478581 8661 webserver.cc:460] Webserver started at http://127.23.100.129:41115/ using document root <none> and password file <none>
I20260806 22:43:19.479569 8661 fs_manager.cc:359] Metadata directory not provided
I20260806 22:43:19.479773 8661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:43:19.480162 8661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 22:43:19 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056199, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:43:19.510138 8661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "de525c1dd5434881a1685e896125a9c9"
format_stamp: "Formatted at 2026-08-06 22:43:19 on dist-test-slave-zr1t"
server_key: "8ece9e1d2707ad88a7d1c5f8d4a79375"
server_key_iv: "754e9b6ee3c7779508abaf3261b4f95a"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:19.515657 8661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "de525c1dd5434881a1685e896125a9c9"
format_stamp: "Formatted at 2026-08-06 22:43:19 on dist-test-slave-zr1t"
server_key: "8ece9e1d2707ad88a7d1c5f8d4a79375"
server_key_iv: "754e9b6ee3c7779508abaf3261b4f95a"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:19.524634 8661 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.003s
I20260806 22:43:19.543932 8689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:43:19.545938 8661 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 22:43:19.546147 8661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "de525c1dd5434881a1685e896125a9c9"
format_stamp: "Formatted at 2026-08-06 22:43:19 on dist-test-slave-zr1t"
server_key: "8ece9e1d2707ad88a7d1c5f8d4a79375"
server_key_iv: "754e9b6ee3c7779508abaf3261b4f95a"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:19.546438 8661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:43:19.573778 8661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:43:19.579151 8661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:43:19.579573 8661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:43:19.580982 8661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:43:19.582720 8661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:43:19.582836 8661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:43:19.582978 8661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:43:19.583073 8661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 22:43:19.624164 8661 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:43909
I20260806 22:43:19.624182 8868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:43909 every 8 connection(s)
I20260806 22:43:19.625728 8661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:43:19.627307 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 8661
I20260806 22:43:19.627571 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:43:19.628635 23954 external_mini_cluster.cc:1428] Setting key a4e4b4370d2d87a28dfbefd2fe8db95f
Aug 06 22:43:19 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056199, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:43:19.682917 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:19.632000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:34371 (local address 127.23.100.190:39035)
0806 22:43:19.655359 (+ 23359us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:19.655378 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:19.659399 (+ 4021us) server_negotiation.cc:407] Connection header received
0806 22:43:19.662859 (+ 3460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:19.662866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:19.662987 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:19.663117 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:43:19.665602 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:19.666784 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:19.669391 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:19.670104 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:19.670220 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:19.675056 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:43:19.675064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:43:19.675069 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:43:19.675113 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:43:19.678869 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:19.679870 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:19.679876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:19.679880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:19.680044 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:19.680621 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:19.680627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:19.680630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:19.681017 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:43:19.681153 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:19.682211 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0806 22:43:19.682487 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23207,"thread_start_us":23074,"threads_started":1}
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
I20260806 22:43:19.685276 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:19.632900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:39035 (local address 127.23.100.129:34371)
0806 22:43:19.659126 (+ 26226us) negotiation.cc:107] Waiting for socket to connect
0806 22:43:19.659203 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 22:43:19.662420 (+ 3217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:43:19.663631 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:43:19.663645 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:43:19.664381 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:43:19.665327 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:19.665360 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:19.667030 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:19.667036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:19.669208 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:19.669225 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:19.670592 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:19.670600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:19.670922 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:19.671811 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:43:19.671856 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:43:19.674736 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:43:19.679073 (+ 4337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:43:19.679082 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:43:19.679101 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:43:19.679710 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:43:19.680207 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:43:19.680212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:43:19.680215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:43:19.680479 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:43:19.681254 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:43:19.681262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:43:19.681745 (+ 483us) client_negotiation.cc:769] Sending connection context
0806 22:43:19.683239 (+ 1494us) client_negotiation.cc:240] Negotiation successful
0806 22:43:19.683800 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25806,"thread_start_us":25480,"threads_started":1}
I20260806 22:43:19.687170 8869 heartbeater.cc:347] Connected to a master server at 127.23.100.190:39035
I20260806 22:43:19.687525 8869 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
I20260806 22:43:19.688479 8869 heartbeater.cc:511] Master 127.23.100.190:39035 requested a full tablet report, sending...
I20260806 22:43:19.691309 8539 ts_manager.cc:194] Registered new tserver with Master: de525c1dd5434881a1685e896125a9c9 (127.23.100.129:43909)
I20260806 22:43:19.693394 8539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:34371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:43:19.730886 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:52773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39035
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:43:19 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056199, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
Aug 06 22:43:20 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056199, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
W20260806 22:43:20.188542 8880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:43:20.189039 8880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:43:20.189244 8880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:43:20.199481 8880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:43:20.199645 8880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:43:20.199900 8880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:43:20.215004 8880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.100.148:52773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:43:20.220005 8880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:43:20.221942 8880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:43:20.283054 8889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:43:20.287580 8891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:43:20.309619 8880 server_base.cc:1034] running on GCE node
W20260806 22:43:20.309813 8897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:43:20.314392 8880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:43:20.324545 8880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:43:20.325898 8880 hybrid_clock.cc:648] HybridClock initialized: now 1786056200325772 us; error 122 us; skew 500 ppm
I20260806 22:43:20.326388 8880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:43:20 dist-test-slave-zr1t krb5kdc[7826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056200, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:43:20.330933 8880 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:43:20.347054 8880 webserver.cc:460] Webserver started at http://127.23.100.130:45913/ using document root <none> and password file <none>
I20260806 22:43:20.348209 8880 fs_manager.cc:359] Metadata directory not provided
I20260806 22:43:20.348426 8880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:43:20.348886 8880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 22:43:20 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056200, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:43:20.369735 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "5fd541b03b724b91b5cb0349b78ec943"
format_stamp: "Formatted at 2026-08-06 22:43:20 on dist-test-slave-zr1t"
server_key: "0b65517ad1a3757f4d7e396282fbb281"
server_key_iv: "74ef94b6985c0c788b80b48c7198b1d6"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:20.371054 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "5fd541b03b724b91b5cb0349b78ec943"
format_stamp: "Formatted at 2026-08-06 22:43:20 on dist-test-slave-zr1t"
server_key: "0b65517ad1a3757f4d7e396282fbb281"
server_key_iv: "74ef94b6985c0c788b80b48c7198b1d6"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:20.385260 8880 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.012s sys 0.000s
I20260806 22:43:20.411372 8906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:43:20.413357 8880 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 22:43:20.413647 8880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "5fd541b03b724b91b5cb0349b78ec943"
format_stamp: "Formatted at 2026-08-06 22:43:20 on dist-test-slave-zr1t"
server_key: "0b65517ad1a3757f4d7e396282fbb281"
server_key_iv: "74ef94b6985c0c788b80b48c7198b1d6"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:20.414031 8880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:43:20.454336 8880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:43:20.459436 8880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:43:20.459792 8880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:43:20.461000 8880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:43:20.462598 8880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:43:20.462708 8880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:43:20.462857 8880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:43:20.462929 8880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:43:20.500670 8880 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:45081
I20260806 22:43:20.500767 9100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:45081 every 8 connection(s)
I20260806 22:43:20.502104 8880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:43:20.508466 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 8880
I20260806 22:43:20.508692 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:43:20.509375 23954 external_mini_cluster.cc:1428] Setting key 214f7b50fb895f5567541348a8d198ab
Aug 06 22:43:20 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056200, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:43:20.527684 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:20.505111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:54929 (local address 127.23.100.190:39035)
0806 22:43:20.505642 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:20.505648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:20.506809 (+ 1161us) server_negotiation.cc:407] Connection header received
0806 22:43:20.508743 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:20.508749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:20.508841 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:20.508990 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:43:20.510681 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:20.511902 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:20.514661 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:20.515298 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:20.515443 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:20.520436 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:43:20.520443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:43:20.520448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:43:20.520491 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:43:20.524105 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:20.524987 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:20.524998 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:20.525000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:20.525219 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:20.525757 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:20.525766 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:20.525768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:20.526157 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:43:20.526332 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:20.527048 (+ 716us) server_negotiation.cc:299] Negotiation successful
0806 22:43:20.527282 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":244,"threads_started":1}
I20260806 22:43:20.528962 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:20.505792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:39035 (local address 127.23.100.130:54929)
0806 22:43:20.506615 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0806 22:43:20.506669 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 22:43:20.508439 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:43:20.509219 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:43:20.509227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:43:20.509765 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:43:20.510487 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:20.510508 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:20.512100 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:20.512107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:20.514473 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:20.514493 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:20.515568 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:20.515577 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:20.515919 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:20.516826 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:43:20.516864 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:43:20.520108 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:43:20.524279 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:43:20.524292 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:43:20.524315 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:43:20.524834 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:43:20.525343 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:43:20.525349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:43:20.525353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:43:20.525633 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:43:20.526291 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:43:20.526301 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:43:20.526674 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 22:43:20.526955 (+ 281us) client_negotiation.cc:240] Negotiation successful
0806 22:43:20.527500 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":363,"threads_started":1}
I20260806 22:43:20.530951 9102 heartbeater.cc:347] Connected to a master server at 127.23.100.190:39035
I20260806 22:43:20.531306 9102 heartbeater.cc:464] Registering TS with master...
I20260806 22:43:20.532176 9102 heartbeater.cc:511] Master 127.23.100.190:39035 requested a full tablet report, sending...
I20260806 22:43:20.533860 8539 ts_manager.cc:194] Registered new tserver with Master: 5fd541b03b724b91b5cb0349b78ec943 (127.23.100.130:45081)
I20260806 22:43:20.534693 8539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:54929
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:43:20.592502 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:52773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39035
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:43:20.697328 8869 heartbeater.cc:503] Master 127.23.100.190:39035 was elected leader, sending a full tablet report...
W20260806 22:43:20.999055 9109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:43:20.999409 9109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:43:20.999581 9109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:43:21.006984 9109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:43:21.007148 9109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:43:21.007341 9109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:43:21.017055 9109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.100.148:52773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:39035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:43:21.018467 9109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:43:21.019907 9109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:43:21.034049 9118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:43:21.034161 9120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:43:21.034082 9116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:43:21.035387 9109 server_base.cc:1034] running on GCE node
I20260806 22:43:21.036036 9109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:43:21.037089 9109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:43:21.038318 9109 hybrid_clock.cc:648] HybridClock initialized: now 1786056201038270 us; error 80 us; skew 500 ppm
I20260806 22:43:21.038686 9109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:43:21 dist-test-slave-zr1t krb5kdc[7826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056201, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:43:21.042191 9109 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:43:21.048557 9109 webserver.cc:460] Webserver started at http://127.23.100.131:38237/ using document root <none> and password file <none>
I20260806 22:43:21.049692 9109 fs_manager.cc:359] Metadata directory not provided
I20260806 22:43:21.049863 9109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:43:21.050328 9109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 22:43:21 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056201, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:43:21.067986 9109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "b7f88eac2e2946479ea9e2b40e0c3a10"
format_stamp: "Formatted at 2026-08-06 22:43:21 on dist-test-slave-zr1t"
server_key: "f621a9ee4c106490169bffca9bffc513"
server_key_iv: "dbfb5a762a876f279c3e50d450338a40"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:21.068871 9109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "b7f88eac2e2946479ea9e2b40e0c3a10"
format_stamp: "Formatted at 2026-08-06 22:43:21 on dist-test-slave-zr1t"
server_key: "f621a9ee4c106490169bffca9bffc513"
server_key_iv: "dbfb5a762a876f279c3e50d450338a40"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:21.074432 9109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 22:43:21.090137 9131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:43:21.091915 9109 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 22:43:21.092130 9109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "b7f88eac2e2946479ea9e2b40e0c3a10"
format_stamp: "Formatted at 2026-08-06 22:43:21 on dist-test-slave-zr1t"
server_key: "f621a9ee4c106490169bffca9bffc513"
server_key_iv: "dbfb5a762a876f279c3e50d450338a40"
server_key_version: "kuduclusterkey@0"
I20260806 22:43:21.092377 9109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:43:21.125248 9109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:43:21.129318 9109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:43:21.129674 9109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:43:21.130992 9109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:43:21.132782 9109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:43:21.132867 9109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:43:21.132954 9109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:43:21.133001 9109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:43:21.168458 9109 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:36929
I20260806 22:43:21.168510 9290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:36929 every 8 connection(s)
I20260806 22:43:21.169943 9109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:43:21.180080 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 9109
I20260806 22:43:21.180279 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:43:21.180775 23954 external_mini_cluster.cc:1428] Setting key dc0b83c4663a4eba3cb1d5e0b1d5ef39
Aug 06 22:43:21 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056201, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:43:21.191846 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.172695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:40573 (local address 127.23.100.190:39035)
0806 22:43:21.173270 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:21.173274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:21.174174 (+ 900us) server_negotiation.cc:407] Connection header received
0806 22:43:21.175976 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:21.175981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:21.176088 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:21.176216 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:43:21.177728 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.178464 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.180346 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.180825 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.180980 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.185165 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:43:21.185170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:43:21.185173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:43:21.185206 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:43:21.188259 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:21.189246 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:21.189257 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:21.189259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:21.189414 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:21.189932 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:21.189938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:21.189945 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:21.190295 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:43:21.190464 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:21.191208 (+ 744us) server_negotiation.cc:299] Negotiation successful
0806 22:43:21.191453 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":317,"threads_started":1}
I20260806 22:43:21.192817 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.173229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:39035 (local address 127.23.100.131:40573)
0806 22:43:21.173984 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 22:43:21.174022 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 22:43:21.175690 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:43:21.176456 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:43:21.176465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:43:21.176908 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:43:21.177486 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.177502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.178616 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.178622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.180144 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.180161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.181106 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.181113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.181432 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.182332 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:43:21.182411 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:43:21.184942 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:43:21.188436 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:43:21.188448 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:43:21.188470 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:43:21.189107 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:43:21.189562 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:43:21.189568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:43:21.189572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:43:21.189792 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:43:21.190436 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:43:21.190445 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:43:21.190812 (+ 367us) client_negotiation.cc:769] Sending connection context
0806 22:43:21.191097 (+ 285us) client_negotiation.cc:240] Negotiation successful
0806 22:43:21.191662 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":313,"threads_started":1}
I20260806 22:43:21.194386 9291 heartbeater.cc:347] Connected to a master server at 127.23.100.190:39035
I20260806 22:43:21.194711 9291 heartbeater.cc:464] Registering TS with master...
I20260806 22:43:21.195456 9291 heartbeater.cc:511] Master 127.23.100.190:39035 requested a full tablet report, sending...
I20260806 22:43:21.196990 8539 ts_manager.cc:194] Registered new tserver with Master: b7f88eac2e2946479ea9e2b40e0c3a10 (127.23.100.131:36929)
I20260806 22:43:21.197952 8539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:40573
I20260806 22:43:21.209810 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:43:21.229305 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.214916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.23.100.190:39035)
0806 22:43:21.215095 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:21.215099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:21.215747 (+ 648us) server_negotiation.cc:407] Connection header received
0806 22:43:21.215897 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:21.215900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:21.215977 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:21.216061 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:43:21.217688 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.218408 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.219896 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.220646 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.220777 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.224085 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:43:21.224090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:43:21.224093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:43:21.224125 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:43:21.226808 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:21.227722 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:21.227725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:21.227727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:21.227844 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:21.228186 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:21.228190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:21.228192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:21.228444 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:43:21.228638 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:21.228902 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 22:43:21.229076 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 22:43:21.233263 8539 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:43:21.240425 8539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:43:21.277233 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.256411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43909 (local address 127.0.0.1:58508)
0806 22:43:21.269366 (+ 12955us) negotiation.cc:107] Waiting for socket to connect
0806 22:43:21.269383 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:43:21.269510 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:43:21.270400 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:43:21.270404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:43:21.270444 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:43:21.271037 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.271054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.272447 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.272456 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.274579 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.274591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.276172 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.276179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.276385 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.276442 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 22:43:21.276593 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 22:43:21.276803 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12846,"mutex_wait_us":68,"spinlock_wait_cycles":4992,"thread_start_us":8232,"threads_started":1}
I20260806 22:43:21.277576 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.258683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.23.100.129:43909)
0806 22:43:21.266009 (+ 7326us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:21.266016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:21.269538 (+ 3522us) server_negotiation.cc:407] Connection header received
0806 22:43:21.269669 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:21.269678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:21.269907 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:21.270134 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:43:21.271225 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.272278 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.274752 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.275952 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.276177 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.276387 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:21.276994 (+ 607us) server_negotiation.cc:299] Negotiation successful
0806 22:43:21.277175 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7164,"thread_start_us":7004,"threads_started":1}
I20260806 22:43:21.278599 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.253787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:36929 (local address 127.0.0.1:34794)
0806 22:43:21.268799 (+ 15012us) negotiation.cc:107] Waiting for socket to connect
0806 22:43:21.268848 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 22:43:21.269025 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:43:21.269821 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:43:21.269829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:43:21.269857 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:43:21.270239 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.270249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.273100 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.273107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.275430 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.275443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.277708 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.277717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.277903 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.277942 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 22:43:21.278052 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 22:43:21.278189 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14900,"mutex_wait_us":27,"thread_start_us":14751,"threads_started":1}
I20260806 22:43:21.279093 9313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.253801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:45081 (local address 127.0.0.1:53848)
0806 22:43:21.265074 (+ 11273us) negotiation.cc:107] Waiting for socket to connect
0806 22:43:21.265114 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 22:43:21.265293 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:43:21.270390 (+ 5097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:43:21.270395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:43:21.270434 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:43:21.271076 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.271094 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.272870 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.272878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.275365 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.275381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.278289 (+ 2908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.278297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.278480 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.278510 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 22:43:21.278590 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 22:43:21.278724 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11014,"mutex_wait_us":80,"spinlock_wait_cycles":384,"thread_start_us":15223,"threads_started":1}
I20260806 22:43:21.279543 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.254560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.23.100.131:36929)
0806 22:43:21.255491 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:21.255503 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:21.268996 (+ 13493us) server_negotiation.cc:407] Connection header received
0806 22:43:21.269237 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:21.269250 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:21.269575 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:21.269820 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:43:21.270428 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.272952 (+ 2524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.275609 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.277140 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.277330 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.277508 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:21.278995 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0806 22:43:21.279187 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":548,"threads_started":1}
I20260806 22:43:21.280067 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.253830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.23.100.130:45081)
0806 22:43:21.269626 (+ 15796us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:21.269635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:21.269658 (+ 23us) server_negotiation.cc:407] Connection header received
0806 22:43:21.269742 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:21.269748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:21.269928 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:21.270810 (+ 882us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:43:21.271314 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.272729 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.275912 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.277368 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.279376 (+ 2008us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.279495 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:21.279546 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 22:43:21.279666 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15560,"thread_start_us":16235,"threads_started":1}
I20260806 22:43:21.282150 8762 tablet_service.cc:1467] Processing CreateTablet for tablet b295ef08750b43af81d5d3b318f37d32 (DEFAULT_TABLE table=test-table [id=397245a51876492d98fef129c3c6d81f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:43:21.283668 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b295ef08750b43af81d5d3b318f37d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:43:21.283308 9204 tablet_service.cc:1467] Processing CreateTablet for tablet b295ef08750b43af81d5d3b318f37d32 (DEFAULT_TABLE table=test-table [id=397245a51876492d98fef129c3c6d81f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:43:21.283622 8997 tablet_service.cc:1467] Processing CreateTablet for tablet b295ef08750b43af81d5d3b318f37d32 (DEFAULT_TABLE table=test-table [id=397245a51876492d98fef129c3c6d81f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:43:21.284760 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b295ef08750b43af81d5d3b318f37d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:43:21.284847 8997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b295ef08750b43af81d5d3b318f37d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:43:21.296375 9316 tablet_bootstrap.cc:492] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: Bootstrap starting.
I20260806 22:43:21.300074 9316 tablet_bootstrap.cc:654] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: Neither blocks nor log segments found. Creating new log.
I20260806 22:43:21.300112 9324 tablet_bootstrap.cc:492] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943: Bootstrap starting.
I20260806 22:43:21.301033 9322 tablet_bootstrap.cc:492] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: Bootstrap starting.
I20260806 22:43:21.301606 9316 log.cc:824] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: Log is configured to *not* fsync() on all Append() calls
I20260806 22:43:21.304559 9324 tablet_bootstrap.cc:654] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943: Neither blocks nor log segments found. Creating new log.
I20260806 22:43:21.304709 9322 tablet_bootstrap.cc:654] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: Neither blocks nor log segments found. Creating new log.
I20260806 22:43:21.305284 9316 tablet_bootstrap.cc:492] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: No bootstrap required, opened a new log
I20260806 22:43:21.305670 9316 ts_tablet_manager.cc:1397] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 22:43:21.306038 9324 log.cc:824] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943: Log is configured to *not* fsync() on all Append() calls
I20260806 22:43:21.306315 9322 log.cc:824] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: Log is configured to *not* fsync() on all Append() calls
I20260806 22:43:21.308902 9324 tablet_bootstrap.cc:492] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943: No bootstrap required, opened a new log
I20260806 22:43:21.309188 9324 ts_tablet_manager.cc:1397] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260806 22:43:21.314133 9322 tablet_bootstrap.cc:492] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: No bootstrap required, opened a new log
I20260806 22:43:21.314450 9322 ts_tablet_manager.cc:1397] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.002s
I20260806 22:43:21.317039 9324 raft_consensus.cc:348] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd541b03b724b91b5cb0349b78ec943" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45081 } } peers { permanent_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 36929 } } peers { permanent_uuid: "de525c1dd5434881a1685e896125a9c9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43909 } }
I20260806 22:43:21.317425 9324 raft_consensus.cc:374] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:43:21.317579 9324 raft_consensus.cc:729] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd541b03b724b91b5cb0349b78ec943, State: Initialized, Role: FOLLOWER
I20260806 22:43:21.317685 9316 raft_consensus.cc:348] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd541b03b724b91b5cb0349b78ec943" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45081 } } peers { permanent_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 36929 } } peers { permanent_uuid: "de525c1dd5434881a1685e896125a9c9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43909 } }
I20260806 22:43:21.318135 9316 raft_consensus.cc:374] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:43:21.318228 9316 raft_consensus.cc:729] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de525c1dd5434881a1685e896125a9c9, State: Initialized, Role: FOLLOWER
I20260806 22:43:21.318282 9324 consensus_queue.cc:260] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd541b03b724b91b5cb0349b78ec943" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45081 } } peers { permanent_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 36929 } } peers { permanent_uuid: "de525c1dd5434881a1685e896125a9c9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43909 } }
I20260806 22:43:21.318915 9316 consensus_queue.cc:260] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd541b03b724b91b5cb0349b78ec943" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45081 } } peers { permanent_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 36929 } } peers { permanent_uuid: "de525c1dd5434881a1685e896125a9c9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43909 } }
I20260806 22:43:21.322310 9102 heartbeater.cc:503] Master 127.23.100.190:39035 was elected leader, sending a full tablet report...
I20260806 22:43:21.323056 9324 ts_tablet_manager.cc:1428] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260806 22:43:21.325023 9316 ts_tablet_manager.cc:1428] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260806 22:43:21.327025 9322 raft_consensus.cc:348] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd541b03b724b91b5cb0349b78ec943" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45081 } } peers { permanent_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 36929 } } peers { permanent_uuid: "de525c1dd5434881a1685e896125a9c9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43909 } }
I20260806 22:43:21.327538 9322 raft_consensus.cc:374] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:43:21.327646 9322 raft_consensus.cc:729] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f88eac2e2946479ea9e2b40e0c3a10, State: Initialized, Role: FOLLOWER
I20260806 22:43:21.328478 9322 consensus_queue.cc:260] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd541b03b724b91b5cb0349b78ec943" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45081 } } peers { permanent_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 36929 } } peers { permanent_uuid: "de525c1dd5434881a1685e896125a9c9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43909 } }
I20260806 22:43:21.329749 9291 heartbeater.cc:503] Master 127.23.100.190:39035 was elected leader, sending a full tablet report...
I20260806 22:43:21.330300 9322 ts_tablet_manager.cc:1428] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260806 22:43:21.361464 9331 raft_consensus.cc:482] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:43:21.361825 9331 raft_consensus.cc:504] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd541b03b724b91b5cb0349b78ec943" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45081 } } peers { permanent_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 36929 } } peers { permanent_uuid: "de525c1dd5434881a1685e896125a9c9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43909 } }
I20260806 22:43:21.363615 9331 leader_election.cc:288] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fd541b03b724b91b5cb0349b78ec943 (127.23.100.130:45081), b7f88eac2e2946479ea9e2b40e0c3a10 (127.23.100.131:36929)
I20260806 22:43:21.372054 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.364287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:54627 (local address 127.23.100.131:36929)
0806 22:43:21.364514 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:21.364520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:21.365477 (+ 957us) server_negotiation.cc:407] Connection header received
0806 22:43:21.365654 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:21.365657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:21.365714 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:21.365784 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:43:21.366323 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.367276 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.369630 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.370891 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.371059 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.371134 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:21.371637 (+ 503us) server_negotiation.cc:299] Negotiation successful
0806 22:43:21.371769 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 22:43:21.372054 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.364043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:33335 (local address 127.23.100.130:45081)
0806 22:43:21.364600 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:21.364609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:21.364818 (+ 209us) server_negotiation.cc:407] Connection header received
0806 22:43:21.364998 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:21.365006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:21.365103 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:21.365207 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:43:21.365915 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.366956 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.369635 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.370887 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.371058 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.371134 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:21.371637 (+ 503us) server_negotiation.cc:299] Negotiation successful
0806 22:43:21.371770 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
I20260806 22:43:21.372416 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.363874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:45081 (local address 127.23.100.129:33335)
0806 22:43:21.364689 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0806 22:43:21.364711 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:43:21.364851 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:43:21.365263 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:43:21.365268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:43:21.365294 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:43:21.365737 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.365747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.367088 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.367092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.369398 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.369416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.371200 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.371205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.371414 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.371493 (+ 79us) client_negotiation.cc:769] Sending connection context
0806 22:43:21.371596 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 22:43:21.371843 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":117,"thread_start_us":640,"threads_started":1}
I20260806 22:43:21.372501 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.364111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:36929 (local address 127.23.100.129:54627)
0806 22:43:21.365337 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0806 22:43:21.365360 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:43:21.365510 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:43:21.365837 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:43:21.365844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:43:21.365871 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:43:21.366172 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.366180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.367416 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.367422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.369471 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:43:21.369485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.371180 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.371189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:43:21.371420 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.371497 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 22:43:21.371595 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 22:43:21.371843 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":81,"thread_start_us":935,"threads_started":1}
I20260806 22:43:21.373606 9033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b295ef08750b43af81d5d3b318f37d32" candidate_uuid: "de525c1dd5434881a1685e896125a9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd541b03b724b91b5cb0349b78ec943" is_pre_election: true
I20260806 22:43:21.373620 9232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b295ef08750b43af81d5d3b318f37d32" candidate_uuid: "de525c1dd5434881a1685e896125a9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" is_pre_election: true
I20260806 22:43:21.374102 9232 raft_consensus.cc:2454] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de525c1dd5434881a1685e896125a9c9 in term 0.
I20260806 22:43:21.374111 9033 raft_consensus.cc:2454] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de525c1dd5434881a1685e896125a9c9 in term 0.
I20260806 22:43:21.374771 8692 leader_election.cc:302] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fd541b03b724b91b5cb0349b78ec943, de525c1dd5434881a1685e896125a9c9; no voters:
I20260806 22:43:21.375173 9331 raft_consensus.cc:2781] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:43:21.375332 9331 raft_consensus.cc:482] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:43:21.375465 9331 raft_consensus.cc:3037] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:43:21.377750 9331 raft_consensus.cc:504] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd541b03b724b91b5cb0349b78ec943" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45081 } } peers { permanent_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 36929 } } peers { permanent_uuid: "de525c1dd5434881a1685e896125a9c9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43909 } }
I20260806 22:43:21.378435 9331 leader_election.cc:288] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [CANDIDATE]: Term 1 election: Requested vote from peers 5fd541b03b724b91b5cb0349b78ec943 (127.23.100.130:45081), b7f88eac2e2946479ea9e2b40e0c3a10 (127.23.100.131:36929)
I20260806 22:43:21.378955 9033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b295ef08750b43af81d5d3b318f37d32" candidate_uuid: "de525c1dd5434881a1685e896125a9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd541b03b724b91b5cb0349b78ec943"
I20260806 22:43:21.378984 9232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b295ef08750b43af81d5d3b318f37d32" candidate_uuid: "de525c1dd5434881a1685e896125a9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10"
I20260806 22:43:21.379171 9033 raft_consensus.cc:3037] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:43:21.379199 9232 raft_consensus.cc:3037] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:43:21.381654 9033 raft_consensus.cc:2454] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de525c1dd5434881a1685e896125a9c9 in term 1.
I20260806 22:43:21.381649 9232 raft_consensus.cc:2454] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de525c1dd5434881a1685e896125a9c9 in term 1.
I20260806 22:43:21.382146 8692 leader_election.cc:302] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fd541b03b724b91b5cb0349b78ec943, de525c1dd5434881a1685e896125a9c9; no voters:
I20260806 22:43:21.382470 9331 raft_consensus.cc:2781] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:43:21.383059 9331 raft_consensus.cc:686] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 1 LEADER]: Becoming Leader. State: Replica: de525c1dd5434881a1685e896125a9c9, State: Running, Role: LEADER
W20260806 22:43:21.383589 8871 tablet.cc:2369] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:43:21.383641 9331 consensus_queue.cc:237] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd541b03b724b91b5cb0349b78ec943" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45081 } } peers { permanent_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 36929 } } peers { permanent_uuid: "de525c1dd5434881a1685e896125a9c9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43909 } }
I20260806 22:43:21.388443 8539 catalog_manager.cc:5495] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to de525c1dd5434881a1685e896125a9c9 (127.23.100.129). New cstate: current_term: 1 leader_uuid: "de525c1dd5434881a1685e896125a9c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd541b03b724b91b5cb0349b78ec943" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 36929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de525c1dd5434881a1685e896125a9c9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43909 } health_report { overall_health: HEALTHY } } }
W20260806 22:43:21.421649 9292 tablet.cc:2369] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:43:21.422725 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.416964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.23.100.129:43909)
0806 22:43:21.417209 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:21.417215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:21.417236 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:43:21.417341 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:21.417345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:21.417432 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:21.417643 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:43:21.418033 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.418776 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.420348 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.420766 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.420926 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.421474 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:43:21.422119 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:43:21.422304 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:21.422373 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 22:43:21.422465 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 22:43:21.430855 9033 raft_consensus.cc:1261] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943 [term 1 FOLLOWER]: Refusing update from remote peer de525c1dd5434881a1685e896125a9c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:43:21.430858 9232 raft_consensus.cc:1261] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10 [term 1 FOLLOWER]: Refusing update from remote peer de525c1dd5434881a1685e896125a9c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:43:21.431922 9335 consensus_queue.cc:1035] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fd541b03b724b91b5cb0349b78ec943" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:43:21.432276 9331 consensus_queue.cc:1035] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f88eac2e2946479ea9e2b40e0c3a10" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:43:21.445966 9343 mvcc.cc:205] Tried to move back new op lower bound from 7315686201050492928 to 7315686200871739392. Current Snapshot: MvccSnapshot[applied={T|T < 7315686201050492928}]
I20260806 22:43:21.463125 8539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:43:21.463402 8539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:43:21.467284 8539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88f8ad1fb7c2482aa684aa8ec806c687: Sending DeleteTablet for 3 replicas of tablet b295ef08750b43af81d5d3b318f37d32
I20260806 22:43:21.468519 8997 tablet_service.cc:1514] Processing DeleteTablet for tablet b295ef08750b43af81d5d3b318f37d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:43:21 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:53848
I20260806 22:43:21.468532 9204 tablet_service.cc:1514] Processing DeleteTablet for tablet b295ef08750b43af81d5d3b318f37d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:43:21 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:34794
I20260806 22:43:21.468755 8762 tablet_service.cc:1514] Processing DeleteTablet for tablet b295ef08750b43af81d5d3b318f37d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:43:21 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:58508
I20260806 22:43:21.471982 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 8661
I20260806 22:43:21.479533 9361 tablet_replica.cc:330] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: stopping tablet replica
I20260806 22:43:21.480767 9361 raft_consensus.cc:2229] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:43:21.482802 9357 tablet_replica.cc:330] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943: stopping tablet replica
I20260806 22:43:21.483271 9361 raft_consensus.cc:2258] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:43:21.484061 9357 raft_consensus.cc:2229] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:43:21.484648 9357 raft_consensus.cc:2258] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:43:21.487110 9357 ts_tablet_manager.cc:1905] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:43:21.487602 9361 ts_tablet_manager.cc:1905] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:43:21.494717 9357 ts_tablet_manager.cc:1918] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:43:21.495002 9357 log.cc:1196] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/b295ef08750b43af81d5d3b318f37d32
I20260806 22:43:21.495625 9357 ts_tablet_manager.cc:1939] T b295ef08750b43af81d5d3b318f37d32 P 5fd541b03b724b91b5cb0349b78ec943: Deleting consensus metadata
I20260806 22:43:21.497766 8522 catalog_manager.cc:4841] TS 5fd541b03b724b91b5cb0349b78ec943 (127.23.100.130:45081): tablet b295ef08750b43af81d5d3b318f37d32 (table test-table [id=397245a51876492d98fef129c3c6d81f]) successfully deleted
I20260806 22:43:21.505820 9361 ts_tablet_manager.cc:1918] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:43:21.506546 9359 tablet_replica.cc:330] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: stopping tablet replica
I20260806 22:43:21.507090 9359 raft_consensus.cc:2229] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:43:21.507687 9359 raft_consensus.cc:2258] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:43:21.508224 9361 log.cc:1196] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/b295ef08750b43af81d5d3b318f37d32
I20260806 22:43:21.508911 9361 ts_tablet_manager.cc:1939] T b295ef08750b43af81d5d3b318f37d32 P de525c1dd5434881a1685e896125a9c9: Deleting consensus metadata
I20260806 22:43:21.510210 9359 ts_tablet_manager.cc:1905] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:43:21.510960 8521 catalog_manager.cc:4841] TS de525c1dd5434881a1685e896125a9c9 (127.23.100.129:43909): tablet b295ef08750b43af81d5d3b318f37d32 (table test-table [id=397245a51876492d98fef129c3c6d81f]) successfully deleted
Aug 06 22:43:21 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:43:21.520519 9359 ts_tablet_manager.cc:1918] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:43:21.520851 9359 log.cc:1196] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/b295ef08750b43af81d5d3b318f37d32
I20260806 22:43:21.521481 9359 ts_tablet_manager.cc:1939] T b295ef08750b43af81d5d3b318f37d32 P b7f88eac2e2946479ea9e2b40e0c3a10: Deleting consensus metadata
I20260806 22:43:21.523569 8524 catalog_manager.cc:4841] TS b7f88eac2e2946479ea9e2b40e0c3a10 (127.23.100.131:36929): tablet b295ef08750b43af81d5d3b318f37d32 (table test-table [id=397245a51876492d98fef129c3c6d81f]) successfully deleted
I20260806 22:43:21.523682 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.473942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.23.100.129:43909)
0806 22:43:21.474197 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:21.474207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:21.490359 (+ 16152us) server_negotiation.cc:407] Connection header received
0806 22:43:21.490946 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:21.490955 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:21.491097 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:21.491200 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:43:21.496852 (+ 5652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.498585 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.506348 (+ 7763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.507073 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.507227 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.515634 (+ 8407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:43:21.515650 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:43:21.515671 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:43:21.515725 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:43:21.519914 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:21.520873 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:21.520883 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:21.520887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:21.521076 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:21.521627 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:21.521632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:21.521636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:21.522107 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:43:21.522405 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:21.523051 (+ 646us) server_negotiation.cc:299] Negotiation successful
0806 22:43:21.523314 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 22:43:21.525291 8864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:43:21.635043 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 8661
I20260806 22:43:21.662932 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 8880
Aug 06 22:43:21 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:43:21.679737 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.663888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.23.100.130:45081)
0806 22:43:21.664076 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:21.664080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:21.664099 (+ 19us) server_negotiation.cc:407] Connection header received
0806 22:43:21.664248 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:21.664252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:21.664367 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:21.664492 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:43:21.665888 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.666718 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.668038 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.668462 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.668574 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.673022 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:43:21.673039 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:43:21.673057 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:43:21.673114 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:43:21.676205 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:21.677075 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:21.677083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:21.677085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:21.677230 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:21.677916 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:21.677923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:21.677927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:21.678383 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:43:21.678737 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:21.679105 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 22:43:21.679358 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 22:43:21.680492 9093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:43:21.773061 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 8880
I20260806 22:43:21.797099 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 9109
Aug 06 22:43:21 dist-test-slave-zr1t krb5kdc[7826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:43:21.814846 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:43:21.798070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.23.100.131:36929)
0806 22:43:21.798244 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 22:43:21.798248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:43:21.798269 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:43:21.799038 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:43:21.799043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:43:21.799129 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:43:21.799253 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:43:21.800754 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.801885 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.803300 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:43:21.803731 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:43:21.803838 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:43:21.808656 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:43:21.808674 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:43:21.808692 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:43:21.808733 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:43:21.811986 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:21.812769 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:21.812777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:21.812779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:21.812911 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:43:21.813255 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:43:21.813259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:43:21.813261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:43:21.813784 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:43:21.814017 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 22:43:21.814324 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 22:43:21.814537 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 22:43:21.815593 9285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:43:21.911070 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 9109
I20260806 22:43:21.934808 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 8495
I20260806 22:43:21.936100 8583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:43:22.050784 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 8495
I20260806 22:43:22.085506 23954 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 22:43:22.431174 23954 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 22:43:22.431473 23954 mini_ranger.cc:66] Stopping Ranger...
I20260806 22:43:22.786654 23954 mini_ranger.cc:68] Stopped Ranger
2026-08-06 22:43:22.786 UTC [7870] LOG: received smart shutdown request
2026-08-06 22:43:22.790 UTC [7870] LOG: background worker "logical replication launcher" (PID 7885) exited with exit code 1
2026-08-06 22:43:22.790 UTC [7877] LOG: shutting down
2026-08-06 22:43:22.939 UTC [7870] LOG: database system is shut down
2026-08-06T22:43:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52063 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:43:23 dist-test-slave-zr1t krb5kdc[9387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:43:23 dist-test-slave-zr1t krb5kdc[9387](info): set up 2 sockets
Aug 06 22:43:23 dist-test-slave-zr1t krb5kdc[9387](info): commencing operation
krb5kdc: starting...
W20260806 22:43:24.469394 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.344s user 0.001s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:43:24 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:43:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:43:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.100.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.100.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/rangeradmin_127.23.100.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/rangeradmin_127.23.100.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.100.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.100.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/rangerlookup_127.23.100.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/rangerlookup_127.23.100.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.100.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.100.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/HTTP_127.23.100.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/HTTP_127.23.100.148@KRBTEST.COM.keytab.
I20260806 22:43:24.882642 23954 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9rdcRI/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/postgres -l logfile start
2026-08-06 22:43:27.380 UTC [9432] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 22:43:27.380 UTC [9432] LOG: listening on IPv4 address "127.23.100.148", port 47773
2026-08-06 22:43:27.383 UTC [9432] LOG: listening on Unix socket "/tmp/.s.PGSQL.47773"
2026-08-06 22:43:27.407 UTC [9435] LOG: database system was shut down at 2026-08-06 22:43:25 UTC
2026-08-06 22:43:27.411 UTC [9432] LOG: database system is ready to accept connections
I20260806 22:43:28.595242 23954 mini_postgres.cc:96] Postgres bound to 47773
2026-08-06 22:43:28.620 UTC [9447] FATAL: database "slave" does not exist
127.23.100.148:47773 - accepting connections
I20260806 22:43:28.621464 23954 mini_ranger.cc:161] Starting Ranger...
I20260806 22:43:28.655761 23954 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 22:43:29.135187 23954 mini_ranger.cc:87] Created ranger Postgres database
I20260806 22:43:29.135296 23954 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-admin
2026-08-06 22:43:29,449 [I] DB FLAVOR :POSTGRES
2026-08-06 22:43:29,449 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 22:43:29,450 [I] Checking connection..
2026-08-06 22:43:29,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 22:43:29,700 [I] Checking connection passed.
2026-08-06 22:43:29,701 [I] --------- Verifying version history table ---------
2026-08-06 22:43:29,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 22:43:29,967 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 22:43:29,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 22:43:30,225 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 22:43:30,225 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 22:43:30,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 22:43:30.490 UTC [9540] WARNING: there is no transaction in progress
2026-08-06 22:43:30,500 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 22:43:30,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 22:43:30,745 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 22:43:30,745 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 22:43:30,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 22:43:30,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 22:43:31,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zr1t.c.gcp-upstream.internal','N') ;"
2026-08-06 22:43:31,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 22:43:31,781 [I] Table x_portal_user does not exist in database ranger
2026-08-06 22:43:31,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 22:43:32,041 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 22:43:32,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 22:43:32,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 22:43:32,581 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 22:43:32,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 22:43:33.586 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.606 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.625 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.643 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.659 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.677 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.735 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.743 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.757 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.767 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.779 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.794 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.805 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.813 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.820 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:33.827 UTC [9697] WARNING: there is no transaction in progress
2026-08-06 22:43:34,530 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 22:43:34,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 22:43:34,780 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 22:43:34,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 22:43:35,062 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 22:43:35,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 22:43:35,322 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 22:43:35,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 22:43:35,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 22:43:35,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zr1t.c.gcp-upstream.internal';"
2026-08-06 22:43:36,099 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 22:43:36,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 22:43:36,384 [I] DB_PATCHES have already been applied
I20260806 22:43:36.391053 23954 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-admin:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9rdcRI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 22:43:36.391157 23954 mini_ranger.cc:193] Using host: 127.23.100.148
I20260806 22:43:36.415817 23954 mini_ranger.cc:227] Ranger admin URL: http://127.23.100.148:59977
Aug 06, 2026 10:43:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:43:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:43:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 10:43:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 10:43:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 10:43:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 10:43:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 22:43:37.939327 23954 mini_ranger.cc:160] Time spent starting Ranger: real 9.318s user 0.002s sys 0.072s
Aug 06, 2026 10:43:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 10:43:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 10:43:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 10:43:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 10:43:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 22:43:59.494626 23954 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.100.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.100.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/rangerkms_127.23.100.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.100.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/rangerkms_127.23.100.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.100.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.100.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.100.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/HTTP_127.23.100.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.100.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/HTTP_127.23.100.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 22:43:59 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056239, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 22:43:59.715467 23954 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 22:43:59.744979 23954 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 22:44:00.585402 23954 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 22:44:00.585566 23954 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-kms
I20260806 22:44:00.585635 23954 mini_ranger_kms.cc:227] Using postgres at 127.23.100.148:47773
2026-08-06 22:44:00,983 [I] DB FLAVOR :POSTGRES
2026-08-06 22:44:00,983 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 22:44:00,984 [I] Checking connection
2026-08-06 22:44:00,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 22:44:01,247 [I] connection success
2026-08-06 22:44:01,247 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 22:44:01,248 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 22:44:01,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 22:44:01,508 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 22:44:01,508 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 22:44:01,508 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 22:44:01,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.100.148:47773/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 22:44:01,789 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 22:44:02.325131 23954 mini_ranger_kms.cc:310] Created kms service
I20260806 22:44:02.560453 23954 mini_ranger_kms.cc:326] Created kudu user
I20260806 22:44:02.606882 23954 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 22:44:02 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056239, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:44:03.035930 23954 mini_ranger_kms.cc:384] Added ranger policy
I20260806 22:44:03.036280 23954 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-kms:/tmp/dist-test-task9rdcRI/build/asan/bin/postgresql.jar:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9rdcRI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 22:44:03.036350 23954 mini_ranger_kms.cc:240] Using host: 127.23.100.148
I20260806 22:44:03.063225 23954 mini_ranger_kms.cc:276] Ranger KMS PID: 9974
I20260806 22:44:03.063385 23954 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.100.148:44933
Aug 06, 2026 10:44:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 10:44:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 10:44:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 10:44:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 10:44:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 10:44:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 22:44:05 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786056245, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:05.910744 23954 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.195s user 0.088s sys 0.354s
I20260806 22:44:05.911019 23954 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 22:44:05.911207 23954 mini_ranger_kms.cc:401] 127.23.100.148:44933/kms/v1/keys
Aug 06 22:44:05 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056245, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.148@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
Aug 06 22:44:06 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056245, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.100.148@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
Aug 06 22:44:08 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:44:08.771360 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:44927
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:44927
--ranger_config_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:44933/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:09.135697 10043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:09.136044 10043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:09.136196 10043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:09.144543 10043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:44:09.144711 10043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:09.144816 10043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:09.144922 10043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:44:09.145007 10043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:44:09.155396 10043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.100.148:44933/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:44927
--ranger_config_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:44927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:09.156728 10043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:09.158174 10043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:09.171761 10050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:09.172263 10051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:09.172411 10054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:09.172832 10043 server_base.cc:1034] running on GCE node
I20260806 22:44:09.173816 10043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:09.175381 10043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:09.176630 10043 hybrid_clock.cc:648] HybridClock initialized: now 1786056249176576 us; error 45 us; skew 500 ppm
I20260806 22:44:09.177026 10043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:09 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056249, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:09.180718 10043 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:44:09.182489 10043 webserver.cc:460] Webserver started at http://127.23.100.190:41167/ using document root <none> and password file <none>
I20260806 22:44:09.183161 10043 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:09.183314 10043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:09.183658 10043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 22:44:09 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056249, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:44:09.249114 10043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "78c49807f6304e018398f904953a0657"
format_stamp: "Formatted at 2026-08-06 22:44:09 on dist-test-slave-zr1t"
server_key: "a457ae70479d5591913a0c44b4a25ca3"
server_key_iv: "61646e418c3759d8868bf93f3bb8792a"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:09.250029 10043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "78c49807f6304e018398f904953a0657"
format_stamp: "Formatted at 2026-08-06 22:44:09 on dist-test-slave-zr1t"
server_key: "a457ae70479d5591913a0c44b4a25ca3"
server_key_iv: "61646e418c3759d8868bf93f3bb8792a"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:09.255890 10043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:44:09.276599 10066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:09.278225 10043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:44:09.278445 10043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "78c49807f6304e018398f904953a0657"
format_stamp: "Formatted at 2026-08-06 22:44:09 on dist-test-slave-zr1t"
server_key: "a457ae70479d5591913a0c44b4a25ca3"
server_key_iv: "61646e418c3759d8868bf93f3bb8792a"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:09.278676 10043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:09.303414 10043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:09.307943 10043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:09.308310 10043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:09.329953 10043 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:44927
I20260806 22:44:09.329973 10140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:44927 every 8 connection(s)
I20260806 22:44:09.331326 10043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:44:09.333905 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 10043
I20260806 22:44:09.334153 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:44:09.335165 23954 external_mini_cluster.cc:1428] Setting key 8e7d845a6db77fbbbb10266e9e887689
I20260806 22:44:09.335688 10141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:09.354569 10141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657: Bootstrap starting.
I20260806 22:44:09.358506 10141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:09.359575 10141 log.cc:824] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:09.362982 10141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657: No bootstrap required, opened a new log
Aug 06 22:44:09 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:09.373594 10141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } }
I20260806 22:44:09.374099 10141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:09.374266 10141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c49807f6304e018398f904953a0657, State: Initialized, Role: FOLLOWER
I20260806 22:44:09.374958 10141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } }
I20260806 22:44:09.375211 10141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:44:09.375365 10141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:44:09.375597 10141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:09.377427 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:09.337469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.23.100.190:44927)
0806 22:44:09.350162 (+ 12693us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:09.350183 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:09.350639 (+ 456us) server_negotiation.cc:407] Connection header received
0806 22:44:09.353275 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:09.353306 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:09.353886 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:09.354345 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:09.356205 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:09.358270 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:09.360342 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:09.361511 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:09.361725 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:09.367050 (+ 5325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:09.367062 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:09.367083 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:09.367138 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:09.371428 (+ 4290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:09.372406 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:09.372416 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:09.372426 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:09.372630 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:09.373126 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:09.373131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:09.373134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:09.373926 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:09.374244 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:09.375533 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0806 22:44:09.375981 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12388,"thread_start_us":12087,"threads_started":1}
I20260806 22:44:09.378141 10141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } }
I20260806 22:44:09.378683 10141 leader_election.cc:302] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78c49807f6304e018398f904953a0657; no voters:
I20260806 22:44:09.379379 10141 leader_election.cc:288] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:44:09.379505 10148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:44:09.382386 10148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 1 LEADER]: Becoming Leader. State: Replica: 78c49807f6304e018398f904953a0657, State: Running, Role: LEADER
I20260806 22:44:09.383095 10148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } }
I20260806 22:44:09.383564 10141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:44:09.386214 10150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78c49807f6304e018398f904953a0657. Latest consensus state: current_term: 1 leader_uuid: "78c49807f6304e018398f904953a0657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } } }
I20260806 22:44:09.386660 10150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:09.387349 10149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78c49807f6304e018398f904953a0657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } } }
I20260806 22:44:09.387696 10155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:44:09.387709 10149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:09.412382 10155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:44:09.436694 10155 catalog_manager.cc:1347] Generated new cluster ID: 869a90ba309b43c2bc48dcb67b699b84
I20260806 22:44:09.436833 10155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:44:09.463469 10155 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:44:09.464599 10141 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 22:44:09.465557 10155 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:44:09.473065 10155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657: Generated new TSK 0
I20260806 22:44:09.474165 10155 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:44:10.353574 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:44933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44927
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:44:10 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786056250, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 22:44:10.792538 10215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:10.793057 10215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:10.793284 10215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:10.805429 10215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:10.805621 10215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:10.805882 10215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:44:10.822234 10215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.100.148:44933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:10.824278 10215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:10.826088 10215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:10.844483 10233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:10.848941 10234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:10.851969 10237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:10.852587 10215 server_base.cc:1034] running on GCE node
I20260806 22:44:10.853449 10215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:10.855159 10215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:10.856427 10215 hybrid_clock.cc:648] HybridClock initialized: now 1786056250856344 us; error 93 us; skew 500 ppm
I20260806 22:44:10.856873 10215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:10 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056250, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:10.864055 10215 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:44:10.875981 10215 webserver.cc:460] Webserver started at http://127.23.100.129:35975/ using document root <none> and password file <none>
I20260806 22:44:10.877005 10215 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:10.877156 10215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:10.877554 10215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 22:44:10 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056250, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:44:10.903244 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "32c9647a6c1a41f0a5b619d9353b4528"
format_stamp: "Formatted at 2026-08-06 22:44:10 on dist-test-slave-zr1t"
server_key: "d43ae24dea3ef7f30c308704be2cb12f"
server_key_iv: "504a20cb40ca8b92c62f1b08e389305d"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:10.904239 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "32c9647a6c1a41f0a5b619d9353b4528"
format_stamp: "Formatted at 2026-08-06 22:44:10 on dist-test-slave-zr1t"
server_key: "d43ae24dea3ef7f30c308704be2cb12f"
server_key_iv: "504a20cb40ca8b92c62f1b08e389305d"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:10.909987 10215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 22:44:10.935235 10245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:10.937193 10215 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260806 22:44:10.937553 10215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "32c9647a6c1a41f0a5b619d9353b4528"
format_stamp: "Formatted at 2026-08-06 22:44:10 on dist-test-slave-zr1t"
server_key: "d43ae24dea3ef7f30c308704be2cb12f"
server_key_iv: "504a20cb40ca8b92c62f1b08e389305d"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:10.937933 10215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:10.974174 10215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:10.980008 10215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:10.980463 10215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:10.987998 10215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:10.996484 10215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:10.996696 10215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 22:44:10.997223 10215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:10.997385 10215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 22:44:11 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056250, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:44:11.502830 10215 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:41677
I20260806 22:44:11.502848 10426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:41677 every 8 connection(s)
I20260806 22:44:11.504289 10215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:44:11.512089 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 10215
I20260806 22:44:11.512328 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:44:11.513191 23954 external_mini_cluster.cc:1428] Setting key fe10c867c014ddd9261aad2e94069b05
Aug 06 22:44:11 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056250, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:11.538399 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:11.507065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:33273 (local address 127.23.100.190:44927)
0806 22:44:11.507648 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:11.507655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:11.508611 (+ 956us) server_negotiation.cc:407] Connection header received
0806 22:44:11.510568 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:11.510575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:11.510671 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:11.510801 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:11.512339 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:11.513146 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:11.515149 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:11.515897 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:11.516060 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:11.521085 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:11.521095 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:11.521103 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:11.521151 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:11.532685 (+ 11534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:11.533867 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:11.533873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:11.533877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:11.534028 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:11.534576 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:11.534581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:11.534583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:11.534975 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:11.535117 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:11.537741 (+ 2624us) server_negotiation.cc:299] Negotiation successful
0806 22:44:11.538006 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":248,"threads_started":1}
I20260806 22:44:11.539305 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:11.507588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:44927 (local address 127.23.100.129:33273)
0806 22:44:11.508427 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:11.508473 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:11.510289 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:11.510997 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:11.511004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:11.511512 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:11.512085 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:11.512101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:11.513320 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:11.513327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:11.514977 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:11.514991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:11.516175 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:11.516183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:11.516520 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:11.517422 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:11.517459 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:11.520780 (+ 3321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:11.532928 (+ 12148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:11.532939 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:11.532959 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:11.533690 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:11.534167 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:11.534173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:11.534176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:11.534436 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:11.535193 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:11.535202 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:11.535665 (+ 463us) client_negotiation.cc:769] Sending connection context
0806 22:44:11.536020 (+ 355us) client_negotiation.cc:240] Negotiation successful
0806 22:44:11.536589 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":364,"threads_started":1}
I20260806 22:44:11.541337 10428 heartbeater.cc:347] Connected to a master server at 127.23.100.190:44927
I20260806 22:44:11.541697 10428 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:11.542423 10428 heartbeater.cc:511] Master 127.23.100.190:44927 requested a full tablet report, sending...
I20260806 22:44:11.544895 10089 ts_manager.cc:194] Registered new tserver with Master: 32c9647a6c1a41f0a5b619d9353b4528 (127.23.100.129:41677)
I20260806 22:44:11.547171 10089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:33273
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:44:11.592870 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:44933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44927
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:44:11 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056250, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
W20260806 22:44:12.057730 10438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:12.058033 10438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:12.058128 10438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:12.065536 10438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:12.065649 10438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:12.065829 10438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:44:12.075929 10438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.100.148:44933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:12.077258 10438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:12.078616 10438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:12.092248 10457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:12.092396 10458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:12.092432 10462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:12.093044 10438 server_base.cc:1034] running on GCE node
I20260806 22:44:12.093818 10438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:12.094820 10438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:12.096024 10438 hybrid_clock.cc:648] HybridClock initialized: now 1786056252095982 us; error 35 us; skew 500 ppm
I20260806 22:44:12.096354 10438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:12 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:12.104866 10438 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:44:12.106494 10438 webserver.cc:460] Webserver started at http://127.23.100.130:37259/ using document root <none> and password file <none>
I20260806 22:44:12.107151 10438 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:12.107256 10438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:12.107522 10438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 22:44:12 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:44:12.125712 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "2b0d29a94712488d9e6d85a8a648893d"
format_stamp: "Formatted at 2026-08-06 22:44:12 on dist-test-slave-zr1t"
server_key: "705c5e4569a7b899b88b068832de6cc9"
server_key_iv: "80aa856b243f83dfaa4fb67cffd57bf4"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:12.126724 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "2b0d29a94712488d9e6d85a8a648893d"
format_stamp: "Formatted at 2026-08-06 22:44:12 on dist-test-slave-zr1t"
server_key: "705c5e4569a7b899b88b068832de6cc9"
server_key_iv: "80aa856b243f83dfaa4fb67cffd57bf4"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:12.135499 10438 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20260806 22:44:12.153414 10469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:12.154927 10438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:44:12.155077 10438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "2b0d29a94712488d9e6d85a8a648893d"
format_stamp: "Formatted at 2026-08-06 22:44:12 on dist-test-slave-zr1t"
server_key: "705c5e4569a7b899b88b068832de6cc9"
server_key_iv: "80aa856b243f83dfaa4fb67cffd57bf4"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:12.155278 10438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:12.178648 10438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:12.182979 10438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:12.183310 10438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:12.184448 10438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:12.186031 10438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:12.186153 10438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:12.186280 10438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:12.186331 10438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:12.222008 10438 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:46119
I20260806 22:44:12.222028 10645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:46119 every 8 connection(s)
I20260806 22:44:12.223426 10438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:44:12.227269 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 10438
I20260806 22:44:12.227470 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:44:12.228015 23954 external_mini_cluster.cc:1428] Setting key 5a76746f438d92b392a12ca218f446e3
Aug 06 22:44:12 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:12.244704 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:12.226183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:59997 (local address 127.23.100.190:44927)
0806 22:44:12.226667 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:12.226672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:12.227609 (+ 937us) server_negotiation.cc:407] Connection header received
0806 22:44:12.229819 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:12.229823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:12.229903 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:12.230019 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:12.231589 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.232312 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.234089 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.234504 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.234644 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:12.238259 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:12.238264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:12.238268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:12.238317 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:12.241233 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:12.242124 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:12.242134 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:12.242137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:12.242270 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:12.242799 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:12.242803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:12.242806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:12.243099 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:12.243220 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:12.244228 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0806 22:44:12.244434 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":257,"threads_started":1}
I20260806 22:44:12.246222 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:12.226595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:44927 (local address 127.23.100.130:59997)
0806 22:44:12.227351 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:12.227441 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:12.229349 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:12.230269 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:12.230278 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:12.230703 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:12.231371 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:12.231396 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.232502 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.232507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:12.233928 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:12.233941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.234746 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.234753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:12.234997 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:12.235715 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:12.235743 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:12.238014 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:12.241453 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:12.241461 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:12.241476 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:12.241984 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:12.242394 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:12.242399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:12.242403 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:12.242650 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:12.243237 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:12.243247 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:12.243740 (+ 493us) client_negotiation.cc:769] Sending connection context
0806 22:44:12.244063 (+ 323us) client_negotiation.cc:240] Negotiation successful
0806 22:44:12.244666 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":348,"threads_started":1}
I20260806 22:44:12.247802 10650 heartbeater.cc:347] Connected to a master server at 127.23.100.190:44927
I20260806 22:44:12.248129 10650 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:12.249053 10650 heartbeater.cc:511] Master 127.23.100.190:44927 requested a full tablet report, sending...
I20260806 22:44:12.250679 10089 ts_manager.cc:194] Registered new tserver with Master: 2b0d29a94712488d9e6d85a8a648893d (127.23.100.130:46119)
I20260806 22:44:12.251544 10089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:59997
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:44:12.297035 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:44933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44927
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:44:12.550901 10428 heartbeater.cc:503] Master 127.23.100.190:44927 was elected leader, sending a full tablet report...
W20260806 22:44:12.655195 10660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:12.655534 10660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:12.655706 10660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:12.663213 10660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:12.663370 10660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:12.663558 10660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:44:12.673548 10660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.100.148:44933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:44927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:12.674923 10660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:12.676240 10660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:12.690125 10671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:12.690120 10673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:12.690068 10675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:12.691216 10660 server_base.cc:1034] running on GCE node
I20260806 22:44:12.691927 10660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:12.692968 10660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:12.694226 10660 hybrid_clock.cc:648] HybridClock initialized: now 1786056252694182 us; error 53 us; skew 500 ppm
I20260806 22:44:12.694603 10660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:12 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:12.697980 10660 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:44:12.699646 10660 webserver.cc:460] Webserver started at http://127.23.100.131:35251/ using document root <none> and password file <none>
I20260806 22:44:12.700552 10660 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:12.700789 10660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:12.701136 10660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 22:44:12 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:44:12.718266 10660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "eebc99be53bd45879b27340de7460e75"
format_stamp: "Formatted at 2026-08-06 22:44:12 on dist-test-slave-zr1t"
server_key: "9689c6880997ded939961f1e70c66ea1"
server_key_iv: "51d1324a9eed4bb054865728511604b1"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:12.719435 10660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "eebc99be53bd45879b27340de7460e75"
format_stamp: "Formatted at 2026-08-06 22:44:12 on dist-test-slave-zr1t"
server_key: "9689c6880997ded939961f1e70c66ea1"
server_key_iv: "51d1324a9eed4bb054865728511604b1"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:12.726846 10660 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260806 22:44:12.744946 10686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:12.746918 10660 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260806 22:44:12.747117 10660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "eebc99be53bd45879b27340de7460e75"
format_stamp: "Formatted at 2026-08-06 22:44:12 on dist-test-slave-zr1t"
server_key: "9689c6880997ded939961f1e70c66ea1"
server_key_iv: "51d1324a9eed4bb054865728511604b1"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:12.747395 10660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:12.766415 10660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:12.770823 10660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:12.771147 10660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:12.772214 10660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:12.773911 10660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:12.773990 10660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:12.774153 10660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:12.774219 10660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:12.810588 10660 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:43205
I20260806 22:44:12.810691 10858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:43205 every 8 connection(s)
I20260806 22:44:12.811939 10660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:44:12.820636 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 10660
I20260806 22:44:12.820838 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:44:12.821327 23954 external_mini_cluster.cc:1428] Setting key bca3eca223bdf4f313bc35345aec448b
Aug 06 22:44:12 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056252, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:12.832522 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:12.814691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:48055 (local address 127.23.100.190:44927)
0806 22:44:12.815190 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:12.815194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:12.816045 (+ 851us) server_negotiation.cc:407] Connection header received
0806 22:44:12.817931 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:12.817936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:12.818049 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:12.818189 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:12.819569 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.820425 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.822425 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.823113 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.823215 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:12.827061 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:12.827066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:12.827069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:12.827102 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:12.829700 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:12.830417 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:12.830421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:12.830424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:12.830547 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:12.830953 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:12.830956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:12.830958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:12.831220 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:12.831333 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:12.832070 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 22:44:12.832240 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
I20260806 22:44:12.833510 10862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:12.815195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:44927 (local address 127.23.100.131:48055)
0806 22:44:12.815880 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:12.815917 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:12.817664 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:12.818425 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:12.818433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:12.818835 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:12.819396 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:12.819412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.820614 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.820618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:12.822233 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:12.822278 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.823372 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.823380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:12.823715 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:12.824537 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:12.824568 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:12.826833 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:12.829848 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:12.829854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:12.829866 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:12.830285 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:12.830641 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:12.830644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:12.830646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:12.830843 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:12.831337 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:12.831343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:12.831737 (+ 394us) client_negotiation.cc:769] Sending connection context
0806 22:44:12.831965 (+ 228us) client_negotiation.cc:240] Negotiation successful
0806 22:44:12.832337 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":284,"threads_started":1}
I20260806 22:44:12.835029 10859 heartbeater.cc:347] Connected to a master server at 127.23.100.190:44927
I20260806 22:44:12.835345 10859 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:12.836236 10859 heartbeater.cc:511] Master 127.23.100.190:44927 requested a full tablet report, sending...
I20260806 22:44:12.837854 10089 ts_manager.cc:194] Registered new tserver with Master: eebc99be53bd45879b27340de7460e75 (127.23.100.131:43205)
I20260806 22:44:12.838699 10089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:48055
I20260806 22:44:12.849227 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:44:12.850972 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 10215
Aug 06 22:44:12 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:44:12.867856 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:12.851931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.23.100.129:41677)
0806 22:44:12.852630 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:12.852637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:12.852661 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:44:12.852717 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:12.852723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:12.852923 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:12.853064 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:12.854732 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.855761 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.857173 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.857815 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.857923 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:12.862184 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:12.862196 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:12.862208 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:12.862252 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:12.865143 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:12.865905 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:12.865911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:12.865914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:12.866052 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:12.866380 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:12.866383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:12.866385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:12.866698 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:12.866950 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:12.867180 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 22:44:12.867500 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":376,"threads_started":1}
I20260806 22:44:12.869082 10422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:12.953459 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 10215
I20260806 22:44:12.976987 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 10438
Aug 06 22:44:12 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:44:12.994200 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:12.977962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.23.100.130:46119)
0806 22:44:12.978557 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:12.978564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:12.978590 (+ 26us) server_negotiation.cc:407] Connection header received
0806 22:44:12.978665 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:12.978672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:12.978872 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:12.979013 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:12.980310 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.981277 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.982841 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:12.983405 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:12.983510 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:12.988244 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:12.988256 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:12.988271 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:12.988306 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:12.991275 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:12.992029 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:12.992036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:12.992039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:12.992174 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:12.992523 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:12.992526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:12.992528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:12.992868 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:12.993102 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:12.993469 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 22:44:12.993775 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":318,"threads_started":1}
I20260806 22:44:12.995504 10640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:13.084019 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 10438
I20260806 22:44:13.111198 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 10660
Aug 06 22:44:13 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:44:13.128118 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:13.112385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.23.100.131:43205)
0806 22:44:13.113056 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:13.113063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:13.113093 (+ 30us) server_negotiation.cc:407] Connection header received
0806 22:44:13.113151 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:13.113158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:13.113366 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:13.113497 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:13.114902 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:13.115914 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:13.117226 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:13.117933 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:13.118037 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:13.122444 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:13.122459 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:13.122475 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:13.122509 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:13.125313 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:13.126125 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:13.126132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:13.126134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:13.126257 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:13.126665 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:13.126669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:13.126671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:13.127029 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:13.127270 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:13.127543 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 22:44:13.127734 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":362,"threads_started":1}
I20260806 22:44:13.129381 10852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:13.213811 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 10660
I20260806 22:44:13.236440 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 10043
I20260806 22:44:13.237828 10134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:13.344780 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 10043
I20260806 22:44:13.374402 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:44927
--webserver_interface=127.23.100.190
--webserver_port=41167
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:44927
--ranger_config_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:44933/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.100.148:44933/kms with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:13.712491 10874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:13.712777 10874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:13.712929 10874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:13.720189 10874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:44:13.720301 10874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:13.720405 10874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:13.720510 10874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:44:13.720595 10874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:44:13.730970 10874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.100.148:44933/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:44927
--ranger_config_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:44927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=41167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:13.732268 10874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:13.733636 10874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:13.745738 10882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:13.745679 10885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:13.745679 10883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:13.746667 10874 server_base.cc:1034] running on GCE node
I20260806 22:44:13.747490 10874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:13.748914 10874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:13.750149 10874 hybrid_clock.cc:648] HybridClock initialized: now 1786056253750097 us; error 63 us; skew 500 ppm
I20260806 22:44:13.750504 10874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:13 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056253, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:13.753906 10874 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:44:13.755405 10874 webserver.cc:460] Webserver started at http://127.23.100.190:41167/ using document root <none> and password file <none>
I20260806 22:44:13.756075 10874 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:13.756233 10874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 22:44:13 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056253, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:44:13.842327 10874 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 22:44:13.845685 10895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:13.847157 10874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:44:13.847399 10874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "78c49807f6304e018398f904953a0657"
format_stamp: "Formatted at 2026-08-06 22:44:09 on dist-test-slave-zr1t"
server_key: "a457ae70479d5591913a0c44b4a25ca3"
server_key_iv: "61646e418c3759d8868bf93f3bb8792a"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:13.848394 10874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:13.863845 10874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:13.867987 10874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:13.868238 10874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:13.887974 10874 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:44927
I20260806 22:44:13.887993 10973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:44927 every 8 connection(s)
I20260806 22:44:13.889708 10874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:44:13.895107 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 10874
I20260806 22:44:13.896130 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:44933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:41677
--local_ip_for_outbound_sockets=127.23.100.129
--tserver_master_addrs=127.23.100.190:44927
--webserver_port=35975
--webserver_interface=127.23.100.129
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.100.148:44933/kms with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:44:13.896908 10974 sys_catalog.cc:263] Verifying existing consensus state
I20260806 22:44:13.900381 10974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657: Bootstrap starting.
I20260806 22:44:13.919863 10974 log.cc:824] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:13.929126 10974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 22:44:13.929770 10974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657: Bootstrap complete.
I20260806 22:44:13.938858 10974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } }
I20260806 22:44:13.940145 10974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c49807f6304e018398f904953a0657, State: Initialized, Role: FOLLOWER
I20260806 22:44:13.940853 10974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } }
I20260806 22:44:13.941108 10974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:44:13.941242 10974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:44:13.941427 10974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 1 FOLLOWER]: Advancing to term 2
I20260806 22:44:13.945256 10974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } }
I20260806 22:44:13.945828 10974 leader_election.cc:302] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78c49807f6304e018398f904953a0657; no voters:
I20260806 22:44:13.946481 10974 leader_election.cc:288] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 22:44:13.946641 10979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 22:44:13.947758 10979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [term 2 LEADER]: Becoming Leader. State: Replica: 78c49807f6304e018398f904953a0657, State: Running, Role: LEADER
I20260806 22:44:13.948352 10979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } }
I20260806 22:44:13.948707 10974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:44:13.951186 10982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78c49807f6304e018398f904953a0657. Latest consensus state: current_term: 2 leader_uuid: "78c49807f6304e018398f904953a0657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } } }
I20260806 22:44:13.951249 10981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "78c49807f6304e018398f904953a0657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c49807f6304e018398f904953a0657" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 44927 } } }
I20260806 22:44:13.951594 10982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:13.951725 10981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:13.955299 10989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:44:13.962230 10989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:44:13.963721 10989 catalog_manager.cc:1259] Loaded cluster ID: 869a90ba309b43c2bc48dcb67b699b84
I20260806 22:44:13.963898 10989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:44:13.967836 10989 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:44:13.969383 10989 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657: Loaded TSK: 0
I20260806 22:44:13.971148 10989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:44:13.973966 10974 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260806 22:44:14.341217 10977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:14.341571 10977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:14.341741 10977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:14.353363 10977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:14.353547 10977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:14.353744 10977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:44:14.369470 10977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.100.148:44933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:41677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=35975
--tserver_master_addrs=127.23.100.190:44927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:14.371313 10977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:14.373230 10977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:14.401602 11032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:14.401633 11031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:14.410894 10977 server_base.cc:1034] running on GCE node
W20260806 22:44:14.414315 11036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:14.415447 10977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:14.424821 10977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:14.428534 10977 hybrid_clock.cc:648] HybridClock initialized: now 1786056254428155 us; error 368 us; skew 500 ppm
I20260806 22:44:14.429118 10977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:14 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056254, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:14.435611 10977 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:44:14.437987 10977 webserver.cc:460] Webserver started at http://127.23.100.129:35975/ using document root <none> and password file <none>
I20260806 22:44:14.439103 10977 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:14.439294 10977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 22:44:14 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056254, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:44:14.467646 10977 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.003s
I20260806 22:44:14.486934 11052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:14.489353 10977 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.008s sys 0.008s
I20260806 22:44:14.489626 10977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "32c9647a6c1a41f0a5b619d9353b4528"
format_stamp: "Formatted at 2026-08-06 22:44:10 on dist-test-slave-zr1t"
server_key: "d43ae24dea3ef7f30c308704be2cb12f"
server_key_iv: "504a20cb40ca8b92c62f1b08e389305d"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:14.491225 10977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:14.546671 10977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:14.552781 10977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:14.553138 10977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:14.554586 10977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:14.561117 10977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:14.561259 10977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:14.561395 10977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:14.561488 10977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:14.616317 10977 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:41677
I20260806 22:44:14.616364 11219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:41677 every 8 connection(s)
I20260806 22:44:14.618016 10977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:44:14.626451 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 10977
I20260806 22:44:14.627482 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:44933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:46119
--local_ip_for_outbound_sockets=127.23.100.130
--tserver_master_addrs=127.23.100.190:44927
--webserver_port=37259
--webserver_interface=127.23.100.130
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.100.148:44933/kms with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:44:14 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056254, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:14.695895 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:14.623644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:44927 (local address 127.23.100.129:52215)
0806 22:44:14.658238 (+ 34594us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:14.658298 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:14.661119 (+ 2821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:14.670196 (+ 9077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:14.670213 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:14.671127 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:14.672141 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:14.672206 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:14.675169 (+ 2963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:14.675177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:14.677659 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:14.677683 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:14.679423 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:14.679434 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:14.679764 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:14.680953 (+ 1189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:14.680990 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:14.684391 (+ 3401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:14.690092 (+ 5701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:14.690106 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:14.690130 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:14.690913 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:14.691572 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:14.691580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:14.691586 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:14.691910 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:14.693112 (+ 1202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:14.693121 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:14.693576 (+ 455us) client_negotiation.cc:769] Sending connection context
0806 22:44:14.693885 (+ 309us) client_negotiation.cc:240] Negotiation successful
0806 22:44:14.694488 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34224,"thread_start_us":33934,"threads_started":1}
I20260806 22:44:14.698503 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:14.625111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:52215 (local address 127.23.100.190:44927)
0806 22:44:14.658192 (+ 33081us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:14.658217 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:14.658471 (+ 254us) server_negotiation.cc:407] Connection header received
0806 22:44:14.662041 (+ 3570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:14.662066 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:14.665648 (+ 3582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:14.666207 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:14.672541 (+ 6334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:14.674874 (+ 2333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:14.677887 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:14.679093 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:14.679362 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:14.684834 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:14.684849 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:14.684874 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:14.684932 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:14.689830 (+ 4898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:14.691136 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:14.691148 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:14.691158 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:14.691402 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:14.692076 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:14.692081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:14.692085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:14.692915 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:14.693159 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:14.696609 (+ 3450us) server_negotiation.cc:299] Negotiation successful
0806 22:44:14.697123 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32728,"thread_start_us":33465,"threads_started":1}
I20260806 22:44:14.702788 11220 heartbeater.cc:347] Connected to a master server at 127.23.100.190:44927
I20260806 22:44:14.703274 11220 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:14.704330 11220 heartbeater.cc:511] Master 127.23.100.190:44927 requested a full tablet report, sending...
W20260806 22:44:14.708618 11220 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:44927 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 22:44:14.709009 11220 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:14.709395 11220 heartbeater.cc:511] Master 127.23.100.190:44927 requested a full tablet report, sending...
I20260806 22:44:14.710896 11220 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:14.711174 11220 heartbeater.cc:511] Master 127.23.100.190:44927 requested a full tablet report, sending...
W20260806 22:44:14.712369 11220 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 22:44:15.114812 11227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:15.115288 11227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:15.115484 11227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:15.127491 11227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:15.127668 11227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:15.127894 11227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:44:15.144486 11227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.100.148:44933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:46119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=37259
--tserver_master_addrs=127.23.100.190:44927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:15.146375 11227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:15.148197 11227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:15.166405 11246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:15.166385 11247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:15.172384 11227 server_base.cc:1034] running on GCE node
W20260806 22:44:15.172819 11251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:15.173753 11227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:15.174989 11227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:15.176263 11227 hybrid_clock.cc:648] HybridClock initialized: now 1786056255176193 us; error 60 us; skew 500 ppm
I20260806 22:44:15.176656 11227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:15 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056255, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:15.180855 11227 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:44:15.187048 11227 webserver.cc:460] Webserver started at http://127.23.100.130:37259/ using document root <none> and password file <none>
I20260806 22:44:15.187971 11227 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:15.188179 11227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 22:44:15 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056255, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:44:15.260097 11227 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.002s sys 0.005s
I20260806 22:44:15.264582 11266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:15.266708 11227 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 22:44:15.266932 11227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "2b0d29a94712488d9e6d85a8a648893d"
format_stamp: "Formatted at 2026-08-06 22:44:12 on dist-test-slave-zr1t"
server_key: "705c5e4569a7b899b88b068832de6cc9"
server_key_iv: "80aa856b243f83dfaa4fb67cffd57bf4"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:15.268577 11227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:15.294394 11227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:15.299860 11227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:15.300220 11227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:15.301582 11227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:15.303676 11227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:15.303819 11227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:15.303953 11227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:15.304049 11227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 22:44:15 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786056255, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:15.374454 11227 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:46119
I20260806 22:44:15.374514 11440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:46119 every 8 connection(s)
I20260806 22:44:15.376405 11227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:44:15.383358 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 11227
I20260806 22:44:15.384464 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.100.148:44933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:43205
--local_ip_for_outbound_sockets=127.23.100.131
--tserver_master_addrs=127.23.100.190:44927
--webserver_port=35251
--webserver_interface=127.23.100.131
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.100.148:44933/kms with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:44:15 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056255, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:15.420670 11444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:15.381632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:56815 (local address 127.23.100.190:44927)
0806 22:44:15.391813 (+ 10181us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:15.391820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:15.393589 (+ 1769us) server_negotiation.cc:407] Connection header received
0806 22:44:15.399431 (+ 5842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:15.399441 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:15.399558 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:15.399697 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:15.401915 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:15.403130 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:15.405672 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:15.406612 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:15.406728 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:15.411748 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:15.411758 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:15.411765 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:15.411814 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:15.416056 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:15.417145 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:15.417154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:15.417162 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:15.417333 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:15.418047 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:15.418055 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:15.418062 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:15.418514 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:15.418676 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:15.419844 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0806 22:44:15.420152 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10025,"thread_start_us":10184,"threads_started":1}
I20260806 22:44:15.423393 11443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:15.380805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:44927 (local address 127.23.100.130:56815)
0806 22:44:15.392520 (+ 11715us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:15.392576 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:15.399013 (+ 6437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:15.400028 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:15.400041 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:15.400714 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:15.401651 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:15.401680 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:15.403389 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:15.403395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:15.405463 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:15.405480 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:15.407151 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:15.407160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:15.407484 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:15.408387 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:15.408418 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:15.411453 (+ 3035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:15.416252 (+ 4799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:15.416265 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:15.416288 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:15.416975 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:15.417555 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:15.417563 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:15.417570 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:15.417891 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:15.418786 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:15.418797 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:15.419318 (+ 521us) client_negotiation.cc:769] Sending connection context
0806 22:44:15.421020 (+ 1702us) client_negotiation.cc:240] Negotiation successful
0806 22:44:15.421740 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11353,"thread_start_us":12572,"threads_started":1}
I20260806 22:44:15.425343 11441 heartbeater.cc:347] Connected to a master server at 127.23.100.190:44927
I20260806 22:44:15.425735 11441 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:15.426790 11441 heartbeater.cc:511] Master 127.23.100.190:44927 requested a full tablet report, sending...
I20260806 22:44:15.428831 10922 ts_manager.cc:194] Registered new tserver with Master: 2b0d29a94712488d9e6d85a8a648893d (127.23.100.130:46119)
I20260806 22:44:15.431577 10922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:56815
I20260806 22:44:15.712530 11220 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:15.713006 11220 heartbeater.cc:511] Master 127.23.100.190:44927 requested a full tablet report, sending...
I20260806 22:44:15.714479 10922 ts_manager.cc:194] Registered new tserver with Master: 32c9647a6c1a41f0a5b619d9353b4528 (127.23.100.129:41677)
I20260806 22:44:15.715638 10922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:52215
Aug 06 22:44:15 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056255, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
W20260806 22:44:15.883119 11445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:15.883550 11445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:15.883715 11445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:15.895660 11445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:15.895812 11445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:15.896000 11445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:44:15.912101 11445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.100.148:44933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:43205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=35251
--tserver_master_addrs=127.23.100.190:44927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:15.913894 11445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:15.915674 11445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:15.935212 11454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:15.935990 11452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:15.940963 11445 server_base.cc:1034] running on GCE node
W20260806 22:44:15.941260 11457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:15.942158 11445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:15.943516 11445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:15.944804 11445 hybrid_clock.cc:648] HybridClock initialized: now 1786056255944736 us; error 46 us; skew 500 ppm
I20260806 22:44:15.945196 11445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:15 dist-test-slave-zr1t krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056255, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:15.949602 11445 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:44:15.953538 11445 webserver.cc:460] Webserver started at http://127.23.100.131:35251/ using document root <none> and password file <none>
I20260806 22:44:15.954525 11445 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:15.954708 11445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 22:44:15 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056255, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:44:15.982686 11445 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:44:15.986647 11468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:15.988592 11445 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 22:44:15.988853 11445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "eebc99be53bd45879b27340de7460e75"
format_stamp: "Formatted at 2026-08-06 22:44:12 on dist-test-slave-zr1t"
server_key: "9689c6880997ded939961f1e70c66ea1"
server_key_iv: "51d1324a9eed4bb054865728511604b1"
server_key_version: "kuduclusterkey@0"
I20260806 22:44:15.990435 11445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:16.011444 11445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:16.016826 11445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:16.017192 11445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:16.018566 11445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:16.020514 11445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:16.020678 11445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:16.020836 11445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:16.020968 11445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:16.076673 11445 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:43205
I20260806 22:44:16.076719 11634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:43205 every 8 connection(s)
I20260806 22:44:16.078663 11445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:44:16.087010 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 11445
Aug 06 22:44:16 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056255, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:16.127753 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.101662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.23.100.190:44927)
0806 22:44:16.106396 (+ 4734us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:16.106402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:16.106423 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:44:16.106488 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:16.106494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:16.106611 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:16.106728 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:16.109352 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.110644 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.115661 (+ 5017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.116204 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.116361 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.120660 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:16.120665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:16.120668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:16.120703 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:16.124295 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:16.125200 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:16.125206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:16.125209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:16.125401 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:16.125898 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:16.125904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:16.125907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:16.126242 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:16.126498 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:16.127122 (+ 624us) server_negotiation.cc:299] Negotiation successful
0806 22:44:16.127372 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4579,"spinlock_wait_cycles":1280,"thread_start_us":5253,"threads_started":1}
I20260806 22:44:16.128693 11639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.085021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:44927 (local address 127.23.100.131:45997)
0806 22:44:16.103539 (+ 18518us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:16.103605 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:16.106064 (+ 2459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:16.107585 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:16.107598 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:16.108210 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:16.109061 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.109087 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.110847 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.110853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.112957 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.112973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.114220 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.114227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.114544 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.115378 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:16.115410 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:16.118090 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:16.122335 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:16.122344 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:16.122364 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:16.122979 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:16.123514 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:16.123519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:16.123523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:16.123755 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:16.125396 (+ 1641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:16.125405 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:16.125916 (+ 511us) client_negotiation.cc:769] Sending connection context
0806 22:44:16.126220 (+ 304us) client_negotiation.cc:240] Negotiation successful
0806 22:44:16.126783 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18176,"thread_start_us":17962,"threads_started":1}
I20260806 22:44:16.129325 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.084417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:45997 (local address 127.23.100.190:44927)
0806 22:44:16.100620 (+ 16203us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:16.100627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:16.104046 (+ 3419us) server_negotiation.cc:407] Connection header received
0806 22:44:16.106537 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:16.106542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:16.106645 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:16.106751 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:16.109255 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.110344 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.113148 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.113804 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.113926 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.118357 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:16.118363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:16.118367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:16.118415 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:16.122143 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:16.123176 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:16.123183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:16.123188 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:16.123364 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:16.123895 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:16.123900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:16.123903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:16.124353 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:16.127357 (+ 3004us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:16.127485 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 22:44:16.127690 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16045,"spinlock_wait_cycles":24832,"thread_start_us":16999,"threads_started":1}
I20260806 22:44:16.131963 11635 heartbeater.cc:347] Connected to a master server at 127.23.100.190:44927
I20260806 22:44:16.132329 11635 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:16.133242 11635 heartbeater.cc:511] Master 127.23.100.190:44927 requested a full tablet report, sending...
I20260806 22:44:16.134948 10919 ts_manager.cc:194] Registered new tserver with Master: eebc99be53bd45879b27340de7460e75 (127.23.100.131:43205)
I20260806 22:44:16.136153 10919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:45997
Aug 06 22:44:16 dist-test-slave-zr1t krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056255, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for HTTP/127.23.100.148@KRBTEST.COM
I20260806 22:44:16.147182 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:44:16.177803 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.151571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.23.100.190:44927)
0806 22:44:16.151845 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:16.151854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:16.154118 (+ 2264us) server_negotiation.cc:407] Connection header received
0806 22:44:16.154328 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:16.154334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:16.154423 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:16.154530 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:16.156329 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.157047 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.158943 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.159652 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.159735 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.163083 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:16.163089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:16.163092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:16.163128 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:16.168695 (+ 5567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:16.173041 (+ 4346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:16.173049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:16.173053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:16.173255 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:16.174779 (+ 1524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:16.174785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:16.174788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:16.175276 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:16.175483 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:16.177058 (+ 1575us) server_negotiation.cc:299] Negotiation successful
0806 22:44:16.177426 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260806 22:44:16.185474 10919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:44:16.198100 10919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:44:16.322685 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.256532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:46119 (local address 127.0.0.1:35052)
0806 22:44:16.312063 (+ 55531us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:16.312122 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:16.312363 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:16.314664 (+ 2301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:16.314673 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:16.314704 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:16.315134 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.315147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.317164 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.317171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.319508 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.319526 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.321687 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.321695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.321911 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.321967 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 22:44:16.322104 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 22:44:16.322322 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55198,"thread_start_us":57980,"threads_started":1}
I20260806 22:44:16.324163 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.257682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.23.100.130:46119)
0806 22:44:16.313011 (+ 55329us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:16.313021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:16.313053 (+ 32us) server_negotiation.cc:407] Connection header received
0806 22:44:16.313164 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:16.313173 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:16.313447 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:16.314392 (+ 945us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:16.315359 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.316986 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.319724 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.321222 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.321401 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.323605 (+ 2204us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:16.323683 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 22:44:16.323875 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55147,"thread_start_us":56992,"threads_started":1}
I20260806 22:44:16.328503 11341 tablet_service.cc:1467] Processing CreateTablet for tablet 1be59cb6321a40a5b80b2f0ca66ba033 (DEFAULT_TABLE table=test-table [id=56ca11c027a34e13ad4d38eb1e05fa1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:16.330158 11341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be59cb6321a40a5b80b2f0ca66ba033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:16.342192 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.269253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:43205 (local address 127.0.0.1:40492)
0806 22:44:16.322993 (+ 53740us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:16.323014 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:16.323119 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:16.334866 (+ 11747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:16.334872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:16.334898 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:16.335086 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.335096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.337231 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.337237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.339159 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.339186 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.341362 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.341369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.341579 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.341614 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 22:44:16.341706 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 22:44:16.341865 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53639,"thread_start_us":64150,"threads_started":1}
I20260806 22:44:16.344224 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.269560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.23.100.131:43205)
0806 22:44:16.334371 (+ 64811us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:16.334378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:16.334405 (+ 27us) server_negotiation.cc:407] Connection header received
0806 22:44:16.334481 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:16.334486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:16.334656 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:16.334869 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:16.335275 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.336990 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.339378 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.340889 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.341078 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.341254 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:16.342810 (+ 1556us) server_negotiation.cc:299] Negotiation successful
0806 22:44:16.343001 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64639,"thread_start_us":64468,"threads_started":1}
I20260806 22:44:16.348902 11544 tablet_service.cc:1467] Processing CreateTablet for tablet 1be59cb6321a40a5b80b2f0ca66ba033 (DEFAULT_TABLE table=test-table [id=56ca11c027a34e13ad4d38eb1e05fa1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:16.350519 11544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be59cb6321a40a5b80b2f0ca66ba033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:16.353662 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.256959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:41677 (local address 127.0.0.1:56492)
0806 22:44:16.314000 (+ 57041us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:16.314021 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:16.314139 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:16.330322 (+ 16183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:16.330328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:16.330359 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:16.330693 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.330707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.347443 (+ 16736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.347451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.349631 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.349649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.351683 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.351690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.351889 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.351927 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 22:44:16.352019 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 22:44:16.353333 (+ 1314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56910,"thread_start_us":56601,"threads_started":1}
I20260806 22:44:16.354214 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.258010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.23.100.129:41677)
0806 22:44:16.327189 (+ 69179us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:16.327202 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:16.327245 (+ 43us) server_negotiation.cc:407] Connection header received
0806 22:44:16.327387 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:16.327398 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:16.327705 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:16.327959 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:16.340330 (+ 12371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.343551 (+ 3221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.349928 (+ 6377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.351414 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.351690 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.351889 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:16.353678 (+ 1789us) server_negotiation.cc:299] Negotiation successful
0806 22:44:16.353884 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68933,"thread_start_us":70188,"threads_started":1}
I20260806 22:44:16.358498 11119 tablet_service.cc:1467] Processing CreateTablet for tablet 1be59cb6321a40a5b80b2f0ca66ba033 (DEFAULT_TABLE table=test-table [id=56ca11c027a34e13ad4d38eb1e05fa1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:16.360464 11119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be59cb6321a40a5b80b2f0ca66ba033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:16.389147 11664 tablet_bootstrap.cc:492] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: Bootstrap starting.
I20260806 22:44:16.394568 11664 tablet_bootstrap.cc:654] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:16.398531 11664 log.cc:824] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:16.401703 11666 tablet_bootstrap.cc:492] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: Bootstrap starting.
I20260806 22:44:16.406922 11665 tablet_bootstrap.cc:492] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528: Bootstrap starting.
I20260806 22:44:16.412436 11664 tablet_bootstrap.cc:492] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: No bootstrap required, opened a new log
I20260806 22:44:16.412811 11664 ts_tablet_manager.cc:1397] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.007s
I20260806 22:44:16.413146 11666 tablet_bootstrap.cc:654] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:16.414499 11666 log.cc:824] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:16.416814 11665 tablet_bootstrap.cc:654] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:16.418097 11665 log.cc:824] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:16.433000 11666 tablet_bootstrap.cc:492] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: No bootstrap required, opened a new log
I20260806 22:44:16.433534 11666 ts_tablet_manager.cc:1397] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.004s
I20260806 22:44:16.436305 11441 heartbeater.cc:503] Master 127.23.100.190:44927 was elected leader, sending a full tablet report...
I20260806 22:44:16.448035 11664 raft_consensus.cc:348] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c9647a6c1a41f0a5b619d9353b4528" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 41677 } } peers { permanent_uuid: "2b0d29a94712488d9e6d85a8a648893d" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46119 } } peers { permanent_uuid: "eebc99be53bd45879b27340de7460e75" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43205 } }
I20260806 22:44:16.449280 11664 raft_consensus.cc:374] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:16.449419 11664 raft_consensus.cc:729] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0d29a94712488d9e6d85a8a648893d, State: Initialized, Role: FOLLOWER
I20260806 22:44:16.450198 11664 consensus_queue.cc:260] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c9647a6c1a41f0a5b619d9353b4528" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 41677 } } peers { permanent_uuid: "2b0d29a94712488d9e6d85a8a648893d" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46119 } } peers { permanent_uuid: "eebc99be53bd45879b27340de7460e75" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43205 } }
I20260806 22:44:16.452643 11665 tablet_bootstrap.cc:492] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528: No bootstrap required, opened a new log
I20260806 22:44:16.453064 11665 ts_tablet_manager.cc:1397] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528: Time spent bootstrapping tablet: real 0.046s user 0.021s sys 0.013s
I20260806 22:44:16.453805 11666 raft_consensus.cc:348] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c9647a6c1a41f0a5b619d9353b4528" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 41677 } } peers { permanent_uuid: "2b0d29a94712488d9e6d85a8a648893d" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46119 } } peers { permanent_uuid: "eebc99be53bd45879b27340de7460e75" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43205 } }
I20260806 22:44:16.454810 11666 raft_consensus.cc:374] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:16.455165 11666 raft_consensus.cc:729] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebc99be53bd45879b27340de7460e75, State: Initialized, Role: FOLLOWER
I20260806 22:44:16.456233 11666 consensus_queue.cc:260] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c9647a6c1a41f0a5b619d9353b4528" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 41677 } } peers { permanent_uuid: "2b0d29a94712488d9e6d85a8a648893d" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46119 } } peers { permanent_uuid: "eebc99be53bd45879b27340de7460e75" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43205 } }
I20260806 22:44:16.461494 11664 ts_tablet_manager.cc:1428] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: Time spent starting tablet: real 0.048s user 0.018s sys 0.001s
I20260806 22:44:16.465763 11635 heartbeater.cc:503] Master 127.23.100.190:44927 was elected leader, sending a full tablet report...
I20260806 22:44:16.466423 11666 ts_tablet_manager.cc:1428] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: Time spent starting tablet: real 0.033s user 0.016s sys 0.005s
I20260806 22:44:16.467401 11665 raft_consensus.cc:348] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c9647a6c1a41f0a5b619d9353b4528" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 41677 } } peers { permanent_uuid: "2b0d29a94712488d9e6d85a8a648893d" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46119 } } peers { permanent_uuid: "eebc99be53bd45879b27340de7460e75" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43205 } }
I20260806 22:44:16.467979 11665 raft_consensus.cc:374] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:16.468122 11665 raft_consensus.cc:729] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c9647a6c1a41f0a5b619d9353b4528, State: Initialized, Role: FOLLOWER
I20260806 22:44:16.468911 11665 consensus_queue.cc:260] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c9647a6c1a41f0a5b619d9353b4528" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 41677 } } peers { permanent_uuid: "2b0d29a94712488d9e6d85a8a648893d" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46119 } } peers { permanent_uuid: "eebc99be53bd45879b27340de7460e75" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43205 } }
I20260806 22:44:16.471421 11220 heartbeater.cc:503] Master 127.23.100.190:44927 was elected leader, sending a full tablet report...
I20260806 22:44:16.473450 11665 ts_tablet_manager.cc:1428] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528: Time spent starting tablet: real 0.020s user 0.010s sys 0.007s
I20260806 22:44:16.553364 11677 raft_consensus.cc:482] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:44:16.553817 11677 raft_consensus.cc:504] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c9647a6c1a41f0a5b619d9353b4528" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 41677 } } peers { permanent_uuid: "2b0d29a94712488d9e6d85a8a648893d" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46119 } } peers { permanent_uuid: "eebc99be53bd45879b27340de7460e75" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43205 } }
I20260806 22:44:16.555773 11677 leader_election.cc:288] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0d29a94712488d9e6d85a8a648893d (127.23.100.130:46119), eebc99be53bd45879b27340de7460e75 (127.23.100.131:43205)
I20260806 22:44:16.565593 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.556818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:54795 (local address 127.23.100.131:43205)
0806 22:44:16.557000 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:16.557006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:16.558689 (+ 1683us) server_negotiation.cc:407] Connection header received
0806 22:44:16.558918 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:16.558924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:16.559012 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:16.559067 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:16.559889 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.560977 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.563628 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.564485 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.564697 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.564778 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:16.565126 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 22:44:16.565273 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:44:16.565699 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.556369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:43205 (local address 127.23.100.129:54795)
0806 22:44:16.558560 (+ 2191us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:16.558581 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:16.558721 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:16.559231 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:16.559235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:16.559255 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:16.559713 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.559728 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.561140 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.561146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.563466 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.563477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.564723 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.564731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.564938 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.564969 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 22:44:16.565085 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 22:44:16.565240 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2026,"mutex_wait_us":1,"thread_start_us":1822,"threads_started":1}
I20260806 22:44:16.565713 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.556250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:57189 (local address 127.23.100.130:46119)
0806 22:44:16.556481 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:16.556487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:16.558230 (+ 1743us) server_negotiation.cc:407] Connection header received
0806 22:44:16.558585 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:16.558592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:16.558690 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:16.558806 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:16.559374 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.560372 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.563214 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.564488 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.564641 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.564716 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:16.565273 (+ 557us) server_negotiation.cc:299] Negotiation successful
0806 22:44:16.565424 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 22:44:16.566447 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.556165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:46119 (local address 127.23.100.129:57189)
0806 22:44:16.558104 (+ 1939us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:16.558126 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:16.558386 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:16.558813 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:16.558818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:16.558848 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:16.559195 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.559207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.560513 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.560521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.563015 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:16.563038 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.564884 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.564891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:16.565075 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.565109 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 22:44:16.565959 (+ 850us) client_negotiation.cc:240] Negotiation successful
0806 22:44:16.566095 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1793,"thread_start_us":1551,"threads_started":1}
I20260806 22:44:16.567093 11572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be59cb6321a40a5b80b2f0ca66ba033" candidate_uuid: "32c9647a6c1a41f0a5b619d9353b4528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebc99be53bd45879b27340de7460e75" is_pre_election: true
I20260806 22:44:16.567775 11572 raft_consensus.cc:2454] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c9647a6c1a41f0a5b619d9353b4528 in term 0.
I20260806 22:44:16.567685 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be59cb6321a40a5b80b2f0ca66ba033" candidate_uuid: "32c9647a6c1a41f0a5b619d9353b4528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0d29a94712488d9e6d85a8a648893d" is_pre_election: true
I20260806 22:44:16.568300 11375 raft_consensus.cc:2454] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c9647a6c1a41f0a5b619d9353b4528 in term 0.
I20260806 22:44:16.568624 11053 leader_election.cc:302] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c9647a6c1a41f0a5b619d9353b4528, eebc99be53bd45879b27340de7460e75; no voters:
I20260806 22:44:16.569135 11677 raft_consensus.cc:2781] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:44:16.569288 11677 raft_consensus.cc:482] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:44:16.569399 11677 raft_consensus.cc:3037] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:16.572549 11677 raft_consensus.cc:504] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c9647a6c1a41f0a5b619d9353b4528" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 41677 } } peers { permanent_uuid: "2b0d29a94712488d9e6d85a8a648893d" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46119 } } peers { permanent_uuid: "eebc99be53bd45879b27340de7460e75" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43205 } }
I20260806 22:44:16.573227 11677 leader_election.cc:288] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [CANDIDATE]: Term 1 election: Requested vote from peers 2b0d29a94712488d9e6d85a8a648893d (127.23.100.130:46119), eebc99be53bd45879b27340de7460e75 (127.23.100.131:43205)
I20260806 22:44:16.573807 11572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be59cb6321a40a5b80b2f0ca66ba033" candidate_uuid: "32c9647a6c1a41f0a5b619d9353b4528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eebc99be53bd45879b27340de7460e75"
I20260806 22:44:16.573813 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be59cb6321a40a5b80b2f0ca66ba033" candidate_uuid: "32c9647a6c1a41f0a5b619d9353b4528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0d29a94712488d9e6d85a8a648893d"
I20260806 22:44:16.574028 11375 raft_consensus.cc:3037] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:16.574020 11572 raft_consensus.cc:3037] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:16.576606 11375 raft_consensus.cc:2454] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c9647a6c1a41f0a5b619d9353b4528 in term 1.
I20260806 22:44:16.576932 11572 raft_consensus.cc:2454] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c9647a6c1a41f0a5b619d9353b4528 in term 1.
I20260806 22:44:16.577082 11058 leader_election.cc:302] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0d29a94712488d9e6d85a8a648893d, 32c9647a6c1a41f0a5b619d9353b4528; no voters:
I20260806 22:44:16.577410 11677 raft_consensus.cc:2781] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:44:16.577890 11677 raft_consensus.cc:686] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 1 LEADER]: Becoming Leader. State: Replica: 32c9647a6c1a41f0a5b619d9353b4528, State: Running, Role: LEADER
I20260806 22:44:16.578464 11677 consensus_queue.cc:237] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c9647a6c1a41f0a5b619d9353b4528" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 41677 } } peers { permanent_uuid: "2b0d29a94712488d9e6d85a8a648893d" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46119 } } peers { permanent_uuid: "eebc99be53bd45879b27340de7460e75" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43205 } }
I20260806 22:44:16.584298 10919 catalog_manager.cc:5495] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32c9647a6c1a41f0a5b619d9353b4528 (127.23.100.129). New cstate: current_term: 1 leader_uuid: "32c9647a6c1a41f0a5b619d9353b4528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32c9647a6c1a41f0a5b619d9353b4528" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b0d29a94712488d9e6d85a8a648893d" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eebc99be53bd45879b27340de7460e75" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43205 } health_report { overall_health: UNKNOWN } } }
W20260806 22:44:16.600241 11637 tablet.cc:2369] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:44:16.644483 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.638220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.23.100.129:41677)
0806 22:44:16.638431 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:16.638437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:16.638543 (+ 106us) server_negotiation.cc:407] Connection header received
0806 22:44:16.638753 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:16.638758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:16.638859 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:16.639017 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:44:16.639390 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.640482 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.641990 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.642634 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.642764 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.643247 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:44:16.643896 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:44:16.644038 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:16.644118 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 22:44:16.644201 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 22:44:16.645404 11442 tablet.cc:2369] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:44:16.654181 11572 raft_consensus.cc:1261] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75 [term 1 FOLLOWER]: Refusing update from remote peer 32c9647a6c1a41f0a5b619d9353b4528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:44:16.654178 11375 raft_consensus.cc:1261] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d [term 1 FOLLOWER]: Refusing update from remote peer 32c9647a6c1a41f0a5b619d9353b4528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:44:16.655243 11677 consensus_queue.cc:1035] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0d29a94712488d9e6d85a8a648893d" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:44:16.655951 11677 consensus_queue.cc:1035] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eebc99be53bd45879b27340de7460e75" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:44:16.684504 11699 mvcc.cc:205] Tried to move back new op lower bound from 7315686427244224512 to 7315686426949881856. Current Snapshot: MvccSnapshot[applied={T|T < 7315686427244224512}]
I20260806 22:44:16.699571 10919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:44:16.699846 10919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:44:16.703326 10919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78c49807f6304e018398f904953a0657: Sending DeleteTablet for 3 replicas of tablet 1be59cb6321a40a5b80b2f0ca66ba033
I20260806 22:44:16.704588 11341 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be59cb6321a40a5b80b2f0ca66ba033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:16 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:35052
I20260806 22:44:16.704798 11119 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be59cb6321a40a5b80b2f0ca66ba033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:16 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:56492
I20260806 22:44:16.706398 11544 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be59cb6321a40a5b80b2f0ca66ba033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:16 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:40492
I20260806 22:44:16.709589 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 10977
I20260806 22:44:16.718247 11712 tablet_replica.cc:330] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: stopping tablet replica
I20260806 22:44:16.718773 11712 raft_consensus.cc:2229] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:44:16.719400 11712 raft_consensus.cc:2258] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:16.721638 11712 ts_tablet_manager.cc:1905] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:16.740597 11712 ts_tablet_manager.cc:1918] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:16.741124 11712 log.cc:1196] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/1be59cb6321a40a5b80b2f0ca66ba033
I20260806 22:44:16.741864 11712 ts_tablet_manager.cc:1939] T 1be59cb6321a40a5b80b2f0ca66ba033 P eebc99be53bd45879b27340de7460e75: Deleting consensus metadata
I20260806 22:44:16.744028 10896 catalog_manager.cc:4841] TS eebc99be53bd45879b27340de7460e75 (127.23.100.131:43205): tablet 1be59cb6321a40a5b80b2f0ca66ba033 (table test-table [id=56ca11c027a34e13ad4d38eb1e05fa1b]) successfully deleted
I20260806 22:44:16.749377 11710 tablet_replica.cc:330] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528: stopping tablet replica
I20260806 22:44:16.754267 11710 raft_consensus.cc:2229] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:44:16.756218 11710 raft_consensus.cc:2258] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:16.763551 11709 tablet_replica.cc:330] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: stopping tablet replica
I20260806 22:44:16.764073 11709 raft_consensus.cc:2229] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:44:16.764575 11709 raft_consensus.cc:2258] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:16.768221 11710 ts_tablet_manager.cc:1905] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:16.769797 11709 ts_tablet_manager.cc:1905] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:16.776500 11710 ts_tablet_manager.cc:1918] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:16.776775 11709 ts_tablet_manager.cc:1918] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:16.777065 11709 log.cc:1196] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/1be59cb6321a40a5b80b2f0ca66ba033
I20260806 22:44:16.777768 11709 ts_tablet_manager.cc:1939] T 1be59cb6321a40a5b80b2f0ca66ba033 P 2b0d29a94712488d9e6d85a8a648893d: Deleting consensus metadata
I20260806 22:44:16.780049 10901 catalog_manager.cc:4841] TS 2b0d29a94712488d9e6d85a8a648893d (127.23.100.130:46119): tablet 1be59cb6321a40a5b80b2f0ca66ba033 (table test-table [id=56ca11c027a34e13ad4d38eb1e05fa1b]) successfully deleted
I20260806 22:44:16.783511 11710 log.cc:1196] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/1be59cb6321a40a5b80b2f0ca66ba033
I20260806 22:44:16.784327 11710 ts_tablet_manager.cc:1939] T 1be59cb6321a40a5b80b2f0ca66ba033 P 32c9647a6c1a41f0a5b619d9353b4528: Deleting consensus metadata
I20260806 22:44:16.786383 10897 catalog_manager.cc:4841] TS 32c9647a6c1a41f0a5b619d9353b4528 (127.23.100.129:41677): tablet 1be59cb6321a40a5b80b2f0ca66ba033 (table test-table [id=56ca11c027a34e13ad4d38eb1e05fa1b]) successfully deleted
I20260806 22:44:16.788998 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.711396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.23.100.129:41677)
0806 22:44:16.711639 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:16.711648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:16.746186 (+ 34538us) server_negotiation.cc:407] Connection header received
0806 22:44:16.746550 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:16.746559 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:16.746678 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:16.746793 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:16.756614 (+ 9821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.762265 (+ 5651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.768528 (+ 6263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.769338 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.769459 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.779863 (+ 10404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:16.779882 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:16.779901 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:16.779952 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:16.784839 (+ 4887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:16.785944 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:16.785954 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:16.785957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:16.786153 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:16.786675 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:16.786682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:16.786686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:16.787201 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:16.787549 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:16.788294 (+ 745us) server_negotiation.cc:299] Negotiation successful
0806 22:44:16.788595 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 22:44:16.790745 11215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:16.928298 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 10977
I20260806 22:44:16.957269 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 11227
I20260806 22:44:16.972978 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:16.958251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.23.100.130:46119)
0806 22:44:16.958462 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:16.958468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:16.958490 (+ 22us) server_negotiation.cc:407] Connection header received
0806 22:44:16.958566 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:16.958571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:16.958646 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:16.958758 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:16.960183 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.961478 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.963360 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:16.964043 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:16.964205 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:16.966944 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:16.966969 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:16.966986 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:16.967024 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:16.970002 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:16.970747 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:16.970755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:16.970757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:16.970899 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:16.971308 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:16.971318 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:16.971320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:16.971707 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:16.972043 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:16.972377 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 22:44:16.972603 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 22:44:16.973989 11431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:17.083124 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 11227
I20260806 22:44:17.108204 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 11445
I20260806 22:44:17.128628 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:17.109118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.23.100.131:43205)
0806 22:44:17.109793 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:17.109798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:17.109817 (+ 19us) server_negotiation.cc:407] Connection header received
0806 22:44:17.109875 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:17.109878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:17.109959 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:17.110064 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:17.111580 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:17.112382 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:17.113753 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:17.114205 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:17.114338 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:17.122017 (+ 7679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:17.122036 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:17.122053 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:17.122100 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:17.125237 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:17.126291 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:17.126299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:17.126302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:17.126441 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:17.126835 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:17.126839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:17.126841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:17.127362 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:17.127714 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:17.128053 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 22:44:17.128249 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":348,"threads_started":1}
I20260806 22:44:17.129472 11628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:17.220911 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 11445
I20260806 22:44:17.246300 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 10874
I20260806 22:44:17.247659 10965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:17.360296 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 10874
I20260806 22:44:17.393373 23954 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 22:44:17.738507 23954 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 22:44:17.738781 23954 mini_ranger.cc:66] Stopping Ranger...
I20260806 22:44:18.084211 23954 mini_ranger.cc:68] Stopped Ranger
2026-08-06 22:44:18.084 UTC [9432] LOG: received smart shutdown request
2026-08-06 22:44:18.087 UTC [9432] LOG: background worker "logical replication launcher" (PID 9445) exited with exit code 1
2026-08-06 22:44:18.087 UTC [9437] LOG: shutting down
2026-08-06 22:44:18.236 UTC [9432] LOG: database system is shut down
2026-08-06T22:44:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55310 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:44:18 dist-test-slave-zr1t krb5kdc[11737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:44:18 dist-test-slave-zr1t krb5kdc[11737](info): set up 2 sockets
Aug 06 22:44:18 dist-test-slave-zr1t krb5kdc[11737](info): commencing operation
krb5kdc: starting...
W20260806 22:44:19.726059 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.003s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:44:19 dist-test-slave-zr1t krb5kdc[11737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:44:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:44:20.012928 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:40265
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:40265,127.23.100.189:35863,127.23.100.188:46195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:20.357204 11757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:20.357594 11757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:20.357757 11757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:20.365154 11757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:44:20.365307 11757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:20.365422 11757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:20.365559 11757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:44:20.365649 11757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:44:20.375774 11757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:40265,127.23.100.189:35863,127.23.100.188:46195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:40265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:20.377162 11757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:20.378588 11757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:20.390683 11767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:20.390730 11765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:20.390936 11764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:20.392113 11757 server_base.cc:1034] running on GCE node
I20260806 22:44:20.392923 11757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:20.394394 11757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:20.395643 11757 hybrid_clock.cc:648] HybridClock initialized: now 1786056260395594 us; error 44 us; skew 500 ppm
I20260806 22:44:20.395996 11757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:20 dist-test-slave-zr1t krb5kdc[11737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:20.399765 11757 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:44:20.401300 11757 webserver.cc:460] Webserver started at http://127.23.100.190:34569/ using document root <none> and password file <none>
I20260806 22:44:20.402051 11757 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:20.402194 11757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:20.402520 11757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:20.405591 11757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "e8442e7642534939ab2d41267936e50f"
format_stamp: "Formatted at 2026-08-06 22:44:20 on dist-test-slave-zr1t"
server_key: "e46a63dfc4b6cc473d6f7097f692c23e"
server_key_iv: "dcc6b3df59d77e5256e6f5b8a1fd2ecf"
server_key_version: "encryptionkey@0"
I20260806 22:44:20.406419 11757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "e8442e7642534939ab2d41267936e50f"
format_stamp: "Formatted at 2026-08-06 22:44:20 on dist-test-slave-zr1t"
server_key: "e46a63dfc4b6cc473d6f7097f692c23e"
server_key_iv: "dcc6b3df59d77e5256e6f5b8a1fd2ecf"
server_key_version: "encryptionkey@0"
I20260806 22:44:20.411985 11757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 22:44:20.415576 11780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:20.416975 11757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:44:20.417160 11757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "e8442e7642534939ab2d41267936e50f"
format_stamp: "Formatted at 2026-08-06 22:44:20 on dist-test-slave-zr1t"
server_key: "e46a63dfc4b6cc473d6f7097f692c23e"
server_key_iv: "dcc6b3df59d77e5256e6f5b8a1fd2ecf"
server_key_version: "encryptionkey@0"
I20260806 22:44:20.417423 11757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:20.438560 11757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:20.443554 11757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:20.443897 11757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:20.463963 11757 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:40265
I20260806 22:44:20.463999 11842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:40265 every 8 connection(s)
I20260806 22:44:20.465351 11757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:44:20.469254 11844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:20.472616 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 11757
I20260806 22:44:20.472867 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:44:20.473742 23954 external_mini_cluster.cc:1428] Setting key ce4049f5ee9ce66d17455abddcb8e814
I20260806 22:44:20.478256 11844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:44:20 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.100.189@KRBTEST.COM; defaulting to no policy
I20260806 22:44:20.505615 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.482352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:40265 (local address 127.0.0.1:47204)
0806 22:44:20.483218 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.483276 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:20.485682 (+ 2406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:20.486781 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:20.486791 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:20.487714 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:20.488259 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:20.488288 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:20.490047 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:20.490055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:20.492101 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:20.492120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:20.493253 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:20.493262 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:20.493571 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:20.494361 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:20.494405 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:20.497310 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:20.500957 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:20.500969 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:20.500982 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:20.501426 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:20.501904 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:20.501911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:20.501916 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:20.502082 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:20.502908 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:20.502916 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:20.503142 (+ 226us) client_negotiation.cc:769] Sending connection context
0806 22:44:20.503426 (+ 284us) client_negotiation.cc:240] Negotiation successful
0806 22:44:20.504063 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":162,"thread_start_us":372,"threads_started":1}
I20260806 22:44:20.505650 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.483694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.23.100.190:40265)
0806 22:44:20.484443 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:20.484454 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:20.484510 (+ 56us) server_negotiation.cc:407] Connection header received
0806 22:44:20.486114 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:20.486125 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:20.486518 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:20.486692 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:20.488515 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:20.489878 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:20.492302 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:20.492985 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:20.493161 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:20.497628 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:20.497638 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:20.497651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:20.497684 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:20.500774 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:20.501594 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:20.501604 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:20.501607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:20.501776 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:20.502203 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:20.502214 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:20.502216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:20.502749 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:20.502985 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:20.503561 (+ 576us) server_negotiation.cc:299] Negotiation successful
0806 22:44:20.503940 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":505,"thread_start_us":335,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.189@KRBTEST.COM" created.
I20260806 22:44:20.509244 11844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } has no permanent_uuid. Determining permanent_uuid...
I20260806 22:44:20.510663 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.510073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37248)
0806 22:44:20.510307 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.510479 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 22:44:20.511046 11786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
I20260806 22:44:20.512955 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.512418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37254)
0806 22:44:20.512628 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.512740 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 22:44:20.513243 11844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:35863: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab.
Entry for principal kudu/127.23.100.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab.
I20260806 22:44:20.542069 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.189:35863
--webserver_interface=127.23.100.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab
--principal=kudu/127.23.100.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:40265,127.23.100.189:35863,127.23.100.188:46195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:44:20.562626 11844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } attempt: 1
I20260806 22:44:20.563967 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.563432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37270)
0806 22:44:20.563637 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.563801 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 22:44:20.565092 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.564728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37276)
0806 22:44:20.564879 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.564958 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 22:44:20.565327 11844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:35863: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
I20260806 22:44:20.633620 11844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } attempt: 2
I20260806 22:44:20.634903 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.634332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37280)
0806 22:44:20.634621 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.634749 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 22:44:20.636121 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.635680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37288)
0806 22:44:20.635900 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.635984 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260806 22:44:20.636404 11844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:35863: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
I20260806 22:44:20.727666 11844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } attempt: 3
I20260806 22:44:20.728907 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.728470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37304)
0806 22:44:20.728665 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.728773 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 22:44:20.730124 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.729726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37320)
0806 22:44:20.729890 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.729967 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 22:44:20.730384 11844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:35863: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
I20260806 22:44:20.873667 11844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } attempt: 4
I20260806 22:44:20.875207 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.874429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37330)
0806 22:44:20.874927 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.875035 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298}
I20260806 22:44:20.876392 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.875953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37346)
0806 22:44:20.876131 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.876245 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 22:44:20.876631 11844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:35863: Network error: Client connection negotiation failed: client connection to 127.23.100.189:35863: connect: Connection refused (error 111)
W20260806 22:44:20.882556 11854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:20.882846 11854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:20.883002 11854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:20.890296 11854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:44:20.890414 11854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:20.890502 11854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:20.890605 11854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:44:20.890673 11854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:44:20.900467 11854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:40265,127.23.100.189:35863,127.23.100.188:46195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab
--principal=kudu/127.23.100.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.189:35863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.100.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:20.901821 11854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:20.903115 11854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:20.914809 11865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:20.915027 11864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:20.915098 11870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:20.915973 11854 server_base.cc:1034] running on GCE node
I20260806 22:44:20.916710 11854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:20.918139 11854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:20.919392 11854 hybrid_clock.cc:648] HybridClock initialized: now 1786056260919351 us; error 60 us; skew 500 ppm
I20260806 22:44:20.919752 11854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:20 dist-test-slave-zr1t krb5kdc[11737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:20.923265 11854 init.cc:377] Logged in from keytab as kudu/127.23.100.189@KRBTEST.COM (short username kudu)
I20260806 22:44:20.924700 11854 webserver.cc:460] Webserver started at http://127.23.100.189:44285/ using document root <none> and password file <none>
I20260806 22:44:20.925393 11854 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:20.925573 11854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:20.925962 11854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:20.928938 11854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/data/instance:
uuid: "a1a12a9884ba4d08aaadee5f4b0c7622"
format_stamp: "Formatted at 2026-08-06 22:44:20 on dist-test-slave-zr1t"
server_key: "bc120b071153a58d9a89d366791c406a"
server_key_iv: "8667c598ee0534ce3a73d514660c74cf"
server_key_version: "encryptionkey@0"
I20260806 22:44:20.929817 11854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/wal/instance:
uuid: "a1a12a9884ba4d08aaadee5f4b0c7622"
format_stamp: "Formatted at 2026-08-06 22:44:20 on dist-test-slave-zr1t"
server_key: "bc120b071153a58d9a89d366791c406a"
server_key_iv: "8667c598ee0534ce3a73d514660c74cf"
server_key_version: "encryptionkey@0"
I20260806 22:44:20.934957 11854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:44:20.938383 11882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:20.939694 11854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:44:20.939915 11854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/wal
uuid: "a1a12a9884ba4d08aaadee5f4b0c7622"
format_stamp: "Formatted at 2026-08-06 22:44:20 on dist-test-slave-zr1t"
server_key: "bc120b071153a58d9a89d366791c406a"
server_key_iv: "8667c598ee0534ce3a73d514660c74cf"
server_key_version: "encryptionkey@0"
I20260806 22:44:20.940111 11854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:20.960659 11854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:20.964784 11854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:20.965085 11854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:20.984052 11854 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.189:35863
I20260806 22:44:20.984138 11963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.189:35863 every 8 connection(s)
I20260806 22:44:20.985405 11854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
I20260806 22:44:20.988962 11965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:20.992244 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 11854
I20260806 22:44:20.992447 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-1/wal/instance
I20260806 22:44:20.992899 23954 external_mini_cluster.cc:1428] Setting key 9638212d3b798fa7b0a3f94c53366a40
I20260806 22:44:20.995354 11965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:21.018009 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.998006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.23.100.190:40265)
0806 22:44:20.998194 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:20.998199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:20.999522 (+ 1323us) server_negotiation.cc:407] Connection header received
0806 22:44:21.001471 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.001477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.001606 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.001711 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.003201 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.003892 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.005471 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.005972 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.006102 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.010361 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.010371 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.010380 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.010431 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.014442 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.015276 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.015285 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.015291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.015456 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.015902 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.015910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.015915 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.016334 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.016481 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.017255 (+ 774us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.017578 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 22:44:21.018730 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:20.998523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:40265 (local address 127.0.0.1:47216)
0806 22:44:20.999362 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:20.999398 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:21.001172 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:21.001962 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:21.001970 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:21.002487 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:21.003020 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.003036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.004056 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.004060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.005322 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.005335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.006201 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.006209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.006548 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.007283 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:21.007318 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:21.010103 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:21.014618 (+ 4515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.014625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.014639 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.015134 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.015599 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.015602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.015605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.015782 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.016488 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:21.016494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:21.016836 (+ 342us) client_negotiation.cc:769] Sending connection context
0806 22:44:21.017079 (+ 243us) client_negotiation.cc:240] Negotiation successful
0806 22:44:21.017499 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":384,"threads_started":1}
I20260806 22:44:21.020552 11965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.100.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.188@KRBTEST.COM" created.
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
I20260806 22:44:21.036876 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.021478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37350)
0806 22:44:21.021673 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.021694 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:21.021819 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:21.022916 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:21.022919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:21.023335 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:21.023478 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.023486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.025239 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.025247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.026246 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.026259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.027365 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.027369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.027678 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.028286 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:21.028313 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:21.030969 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:21.034344 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.034351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.034355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.034739 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.035127 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.035133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.035136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.035266 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.035867 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:21.035871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:21.036025 (+ 154us) client_negotiation.cc:769] Sending connection context
0806 22:44:21.036165 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 22:44:21.036448 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":22}
I20260806 22:44:21.036971 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.021616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.23.100.189:35863)
0806 22:44:21.022362 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.022373 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.022408 (+ 35us) server_negotiation.cc:407] Connection header received
0806 22:44:21.022469 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.022478 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.022721 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.022888 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.023656 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.025022 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.026412 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.027214 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.027364 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.031196 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.031208 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.031222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.031250 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.034198 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.034879 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.034886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.034889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.035021 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.035373 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.035382 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.035384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.035745 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.035947 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.036293 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.036565 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":524,"thread_start_us":370,"threads_started":1}
I20260806 22:44:21.039913 11965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } has no permanent_uuid. Determining permanent_uuid...
I20260806 22:44:21.041204 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.040757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60526)
0806 22:44:21.040932 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.041041 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 22:44:21.041489 11885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
I20260806 22:44:21.043289 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.042831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60532)
0806 22:44:21.043011 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.043130 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260806 22:44:21.043623 11965 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:46195: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab.
Entry for principal kudu/127.23.100.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab.
I20260806 22:44:21.065765 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.188:46195
--webserver_interface=127.23.100.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab
--principal=kudu/127.23.100.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:40265,127.23.100.189:35863,127.23.100.188:46195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:44:21.092979 11965 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } attempt: 1
I20260806 22:44:21.094203 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.093734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60542)
0806 22:44:21.093929 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.094041 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 22:44:21.095177 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.094865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60556)
0806 22:44:21.095009 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.095082 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 22:44:21.095398 11965 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:46195: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
I20260806 22:44:21.163658 11965 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } attempt: 2
I20260806 22:44:21.164837 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.164344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60564)
0806 22:44:21.164586 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.164673 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 22:44:21.165867 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.165495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60580)
0806 22:44:21.165673 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.165747 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260806 22:44:21.166136 11965 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:46195: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
I20260806 22:44:21.175881 11844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } attempt: 5
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
I20260806 22:44:21.190749 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.176623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37366)
0806 22:44:21.176863 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.176886 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:21.177039 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:21.177497 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:21.177500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:21.177936 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:21.178119 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.178127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.179595 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.179599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.180541 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.180554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.181639 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.181644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.181789 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.182428 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:21.182474 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:21.184498 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:21.188314 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.188318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.188321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.188751 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.189224 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.189227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.189229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.189357 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.189990 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:21.189993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:21.190121 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 22:44:21.190248 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 22:44:21.190501 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 22:44:21.191201 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.176904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.23.100.189:35863)
0806 22:44:21.177145 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.177152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.177174 (+ 22us) server_negotiation.cc:407] Connection header received
0806 22:44:21.177240 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.177245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.177322 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.177429 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.178290 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.179435 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.180703 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.181404 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.181576 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.184692 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.184699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.184704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.184756 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.188167 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.188882 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.188890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.188896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.189095 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.189462 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.189467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.189470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.189865 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.190033 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.190388 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.190624 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 22:44:21.192175 11844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } has no permanent_uuid. Determining permanent_uuid...
I20260806 22:44:21.193202 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.192835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60594)
0806 22:44:21.193004 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.193078 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 22:44:21.194201 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.193901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60606)
0806 22:44:21.194045 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.194106 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 22:44:21.194482 11844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:46195: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
I20260806 22:44:21.245784 11844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } attempt: 1
I20260806 22:44:21.247093 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.246599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60614)
0806 22:44:21.246812 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.246923 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 22:44:21.248135 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.247826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60628)
0806 22:44:21.247959 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.248029 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 22:44:21.248446 11844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:46195: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
I20260806 22:44:21.257360 11965 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } attempt: 3
I20260806 22:44:21.258533 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.258103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60636)
0806 22:44:21.258284 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.258373 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 22:44:21.259524 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.259142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60640)
0806 22:44:21.259353 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.259429 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260806 22:44:21.259778 11965 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:46195: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
I20260806 22:44:21.316699 11844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } attempt: 2
I20260806 22:44:21.318013 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.317492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60644)
0806 22:44:21.317753 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.317845 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260806 22:44:21.319442 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.318881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60654)
0806 22:44:21.319230 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.319337 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260806 22:44:21.319710 11844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:46195: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
I20260806 22:44:21.403014 11965 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } attempt: 4
I20260806 22:44:21.404102 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.403631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60660)
0806 22:44:21.403864 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.403958 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 22:44:21.405055 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.404735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60670)
0806 22:44:21.404880 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.404957 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 22:44:21.405325 11965 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:46195: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
W20260806 22:44:21.425505 11974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:21.425839 11974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:21.426011 11974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 22:44:21.425999 11844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } attempt: 3
I20260806 22:44:21.427101 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.426625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60684)
0806 22:44:21.426820 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.426908 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 22:44:21.428335 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.427916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60700)
0806 22:44:21.428136 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.428223 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260806 22:44:21.428647 11844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:46195: Network error: Client connection negotiation failed: client connection to 127.23.100.188:46195: connect: Connection refused (error 111)
W20260806 22:44:21.434198 11974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:44:21.434306 11974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:21.434386 11974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:21.434459 11974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:44:21.434530 11974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:44:21.445636 11974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:40265,127.23.100.189:35863,127.23.100.188:46195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab
--principal=kudu/127.23.100.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.188:46195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.100.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:21.447476 11974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:21.449041 11974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:21.460757 11986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:21.460814 11984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:21.461352 11988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:21.462105 11974 server_base.cc:1034] running on GCE node
I20260806 22:44:21.463135 11974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:21.465095 11974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:21.466332 11974 hybrid_clock.cc:648] HybridClock initialized: now 1786056261466270 us; error 50 us; skew 500 ppm
I20260806 22:44:21.466746 11974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056261, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:21.470775 11974 init.cc:377] Logged in from keytab as kudu/127.23.100.188@KRBTEST.COM (short username kudu)
I20260806 22:44:21.474812 11974 webserver.cc:460] Webserver started at http://127.23.100.188:42437/ using document root <none> and password file <none>
I20260806 22:44:21.475510 11974 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:21.475638 11974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:21.475965 11974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:21.478981 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/data/instance:
uuid: "1c1fca4d95a84ccb952ec24117c9a12e"
format_stamp: "Formatted at 2026-08-06 22:44:21 on dist-test-slave-zr1t"
server_key: "9d3b797cc7ee015dd1dfa228aab9b500"
server_key_iv: "8fbbdbed37b85de7cde97d53f19e9788"
server_key_version: "encryptionkey@0"
I20260806 22:44:21.479808 11974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/wal/instance:
uuid: "1c1fca4d95a84ccb952ec24117c9a12e"
format_stamp: "Formatted at 2026-08-06 22:44:21 on dist-test-slave-zr1t"
server_key: "9d3b797cc7ee015dd1dfa228aab9b500"
server_key_iv: "8fbbdbed37b85de7cde97d53f19e9788"
server_key_version: "encryptionkey@0"
I20260806 22:44:21.485144 11974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 22:44:21.488735 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:21.490135 11974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:44:21.490312 11974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/wal
uuid: "1c1fca4d95a84ccb952ec24117c9a12e"
format_stamp: "Formatted at 2026-08-06 22:44:21 on dist-test-slave-zr1t"
server_key: "9d3b797cc7ee015dd1dfa228aab9b500"
server_key_iv: "8fbbdbed37b85de7cde97d53f19e9788"
server_key_version: "encryptionkey@0"
I20260806 22:44:21.490514 11974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:21.504168 11974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:21.508788 11974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:21.509105 11974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:21.528038 11974 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.188:46195
I20260806 22:44:21.528071 12072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.188:46195 every 8 connection(s)
I20260806 22:44:21.529397 11974 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
I20260806 22:44:21.533252 12073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:21.535418 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 11974
I20260806 22:44:21.535629 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786056046488940-23954-0/minicluster-data/master-2/wal/instance
I20260806 22:44:21.536270 23954 external_mini_cluster.cc:1428] Setting key b7115356edc42b77fbf5880280939f2a
I20260806 22:44:21.543692 12073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056261, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:21.559970 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.538317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.23.100.188:46195)
0806 22:44:21.539164 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.539176 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.539217 (+ 41us) server_negotiation.cc:407] Connection header received
0806 22:44:21.540867 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.540890 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.541298 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.541724 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.543600 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.545043 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.546805 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.547932 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.548154 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.552402 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.552411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.552421 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.552461 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.555842 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.556623 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.556630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.556636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.556787 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.557197 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.557201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.557203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.557771 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.558074 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.558566 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.559003 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":406,"threads_started":1}
I20260806 22:44:21.561559 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.546600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:40265 (local address 127.0.0.1:47230)
0806 22:44:21.547511 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.547560 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:21.547741 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:21.548123 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:21.548129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:21.548737 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:21.548996 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.549005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.550124 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.550133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.551112 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.551128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.551957 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.551968 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.552098 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.552761 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:21.552796 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:21.555447 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:21.558537 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.558550 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.558557 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.559156 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.559547 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.559551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.559555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.559685 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.560405 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:21.560412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:21.560627 (+ 215us) client_negotiation.cc:769] Sending connection context
0806 22:44:21.560842 (+ 215us) client_negotiation.cc:240] Negotiation successful
0806 22:44:21.561152 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":573,"threads_started":1}
I20260806 22:44:21.561579 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.546635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.23.100.190:40265)
0806 22:44:21.547201 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.547210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.547658 (+ 448us) server_negotiation.cc:407] Connection header received
0806 22:44:21.547869 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.547879 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.547943 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.548055 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.549159 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.549978 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.551254 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.551718 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.551841 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.555655 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.555661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.555664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.555697 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.558399 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.559275 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.559282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.559285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.559431 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.559793 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.559801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.559804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.560057 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.560219 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.560978 (+ 759us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.561154 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":280,"threads_started":1}
I20260806 22:44:21.563146 12073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056261, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:21.576848 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.564019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:35863 (local address 127.0.0.1:37368)
0806 22:44:21.564178 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.564197 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:21.564301 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:21.564630 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:21.564635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:21.565007 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:21.565151 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.565158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.566181 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.566188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.567053 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.567064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.567759 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.567764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.567974 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.568530 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:21.568547 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:21.571115 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:21.574128 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.574134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.574138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.574671 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.575151 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.575159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.575166 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.575342 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.576031 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:21.576039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:21.576230 (+ 191us) client_negotiation.cc:769] Sending connection context
0806 22:44:21.576378 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 22:44:21.576614 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 22:44:21.577150 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.563965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.23.100.189:35863)
0806 22:44:21.564161 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.564167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.564246 (+ 79us) server_negotiation.cc:407] Connection header received
0806 22:44:21.564426 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.564432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.564507 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.564608 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.565293 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.566049 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.567196 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.567613 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.567735 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.571297 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.571302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.571305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.571337 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.573997 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.574802 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.574809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.574814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.575028 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.575602 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.575609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.575611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.575896 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.576027 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.576535 (+ 508us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.576758 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 22:44:21.578161 12073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } has no permanent_uuid. Determining permanent_uuid...
I20260806 22:44:21.579262 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.563671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.23.100.190:40265)
0806 22:44:21.563859 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.563865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.563882 (+ 17us) server_negotiation.cc:407] Connection header received
0806 22:44:21.564265 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.564269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.564337 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.564466 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.565713 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.566390 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.567684 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.568096 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.568226 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.573030 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.573035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.573038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.573069 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.576253 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.577153 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.577159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.577165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.577327 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.577766 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.577772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.577777 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.578060 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.578262 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.578682 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.578942 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056261, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
I20260806 22:44:21.593011 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.578965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60732)
0806 22:44:21.579375 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.579394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:21.579508 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:21.579812 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:21.579817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:21.580176 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:21.580386 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.580396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.581606 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.581611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.583060 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.583074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.584024 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.584028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.584189 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.584733 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:21.584758 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:21.587446 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:21.590417 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.590423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.590427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.590944 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.591282 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.591289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.591294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.591439 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.592112 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:21.592119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:21.592352 (+ 233us) client_negotiation.cc:769] Sending connection context
0806 22:44:21.592481 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 22:44:21.592702 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307}
I20260806 22:44:21.593277 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.579213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.23.100.188:46195)
0806 22:44:21.579378 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.579386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.579448 (+ 62us) server_negotiation.cc:407] Connection header received
0806 22:44:21.579620 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.579623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.579678 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.579754 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.580503 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.581440 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.583213 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.583883 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.584006 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.587631 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.587638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.587644 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.587686 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.590285 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.591053 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.591056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.591058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.591162 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.591545 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.591550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.591552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.591910 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.592059 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.592630 (+ 571us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.592877 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 22:44:21.597676 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.581053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.23.100.189:35863)
0806 22:44:21.581207 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.581213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.581253 (+ 40us) server_negotiation.cc:407] Connection header received
0806 22:44:21.581396 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.581402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.581475 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.581613 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.583218 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.584265 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.586199 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.586928 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.587060 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.591384 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.591388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.591391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.591420 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.594914 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.595722 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.595727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.595731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.595947 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.596291 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.596297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.596300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.596640 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.596926 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.597094 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.597323 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 22:44:21.599388 12073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e: Bootstrap starting.
I20260806 22:44:21.602545 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:44:21.603104 12073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:21.604139 12073 log.cc:824] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:21.605902 11844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } attempt: 4
I20260806 22:44:21.607712 12073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e: No bootstrap required, opened a new log
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
I20260806 22:44:21.625883 12073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } }
I20260806 22:44:21.626462 12073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:21.626573 12073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1fca4d95a84ccb952ec24117c9a12e, State: Initialized, Role: FOLLOWER
I20260806 22:44:21.631817 12073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } }
I20260806 22:44:21.635667 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.606601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60746)
0806 22:44:21.606951 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.606974 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:21.607114 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:21.612479 (+ 5365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:21.612488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:21.613209 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:21.613546 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.613564 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.617473 (+ 3909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.617482 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.619396 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.619417 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.621898 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.621908 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.622183 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.623050 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:21.623082 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:21.626570 (+ 3488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:21.632497 (+ 5927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.632505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.632510 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.633089 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.633635 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.633641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.633645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.633825 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.634585 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:21.634590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:21.634780 (+ 190us) client_negotiation.cc:769] Sending connection context
0806 22:44:21.634938 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 22:44:21.635265 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260806 22:44:21.636955 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.606621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.23.100.188:46195)
0806 22:44:21.606845 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.606852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.607084 (+ 232us) server_negotiation.cc:407] Connection header received
0806 22:44:21.607234 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.607239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.607318 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.613815 (+ 6497us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.613992 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.615449 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.619665 (+ 4216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.620590 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.622643 (+ 2053us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.627190 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.627200 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.627207 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.627248 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.631347 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.633270 (+ 1923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.633276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.633280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.633461 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.633984 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.633989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.633992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.634454 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.636084 (+ 1630us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.636241 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.636534 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 22:44:21.643571 11844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f: Bootstrap starting.
I20260806 22:44:21.645061 12099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } } }
I20260806 22:44:21.645823 12099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:44:21.647536 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.607611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.23.100.190:40265)
0806 22:44:21.607799 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.607805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.627675 (+ 19870us) server_negotiation.cc:407] Connection header received
0806 22:44:21.627873 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.627879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.627961 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.628060 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.630732 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.631847 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.634349 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.635021 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.635101 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.639987 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.639996 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.640001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.640040 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.643917 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.644765 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.644770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.644774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.644931 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.645406 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.645413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.645418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.646084 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.646351 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.646936 (+ 585us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.647180 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 22:44:21.648082 11844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:21.649614 11844 log.cc:824] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:21.650190 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.608114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.23.100.189:35863)
0806 22:44:21.608348 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.608357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.632656 (+ 24299us) server_negotiation.cc:407] Connection header received
0806 22:44:21.632824 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.632832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.632925 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.633028 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.635549 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.636642 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.638857 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.639588 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.639701 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.643629 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.643638 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.643642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.643691 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.646864 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.647729 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.647733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.647737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.647913 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.648399 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.648404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.648407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.648775 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.649020 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.649462 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.649804 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 22:44:21.651888 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.608519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.23.100.188:46195)
0806 22:44:21.628547 (+ 20028us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.628559 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.628589 (+ 30us) server_negotiation.cc:407] Connection header received
0806 22:44:21.628656 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.628664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.628753 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.628888 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.632904 (+ 4016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.634213 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.636345 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.637069 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.637185 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.640929 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.640937 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.640943 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.640996 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.647640 (+ 6644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.649472 (+ 1832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.649477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.649480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.649671 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.650160 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.650167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.650172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.650605 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.650834 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.651251 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.651496 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19849,"thread_start_us":19594,"threads_started":1}
I20260806 22:44:21.651919 12073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:44:21.655537 11844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f: No bootstrap required, opened a new log
W20260806 22:44:21.663146 12115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 22:44:21.663283 12115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 22:44:21.669170 11844 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } }
I20260806 22:44:21.669632 11844 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:21.669759 11844 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8442e7642534939ab2d41267936e50f, State: Initialized, Role: FOLLOWER
I20260806 22:44:21.661047 11974 master_runner.cc:186] Error getting master registration for 127.23.100.190:40265: OK, instance_id { permanent_uuid: "e8442e7642534939ab2d41267936e50f" instance_seqno: 1786056260458320 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 22:44:21.670219 11974 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 22:44:21.670456 11844 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } }
I20260806 22:44:21.671371 12119 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } } }
I20260806 22:44:21.671751 12119 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:44:21.672785 11844 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [sys.catalog]: configured and running, proceeding with master startup.
W20260806 22:44:21.685132 12131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 22:44:21.685412 12131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 22:44:21.683269 11757 master_runner.cc:186] Error getting master registration for 127.23.100.189:35863: OK, instance_id { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" instance_seqno: 1786056260978715 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 22:44:21.695772 11757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 22:44:21.697111 11788 tablet.cc:2369] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:44:21.704564 11965 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } attempt: 5
Aug 06 22:44:21 dist-test-slave-zr1t krb5kdc[11737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
I20260806 22:44:21.722565 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.705236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:46195 (local address 127.0.0.1:60758)
0806 22:44:21.705403 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:21.705426 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:21.705571 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:21.706010 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:21.706016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:21.706578 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:21.706800 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.706814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.708278 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.708284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.709793 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:21.709809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.710943 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.710950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:21.711187 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.711923 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:21.711950 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:21.714963 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:21.719510 (+ 4547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.719517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.719521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.720053 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.720498 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:21.720504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:21.720508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:21.720675 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:21.721432 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:21.721437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:21.721670 (+ 233us) client_negotiation.cc:769] Sending connection context
0806 22:44:21.721841 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 22:44:21.722171 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 22:44:21.722978 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:21.705407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.23.100.188:46195)
0806 22:44:21.705642 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:21.705648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:21.705669 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:44:21.705744 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:21.705749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:21.705839 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:21.705939 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:21.706977 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.708128 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.709943 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:21.710632 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:21.710759 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:21.715191 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:21.715199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:21.715204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:21.715244 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:21.719365 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.720190 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.720196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.720200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.720367 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:21.720810 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:21.720816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:21.720819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:21.721183 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:21.721362 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:21.722061 (+ 699us) server_negotiation.cc:299] Negotiation successful
0806 22:44:21.722313 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 22:44:21.729120 11965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622: Bootstrap starting.
I20260806 22:44:21.733382 11965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:21.734771 11965 log.cc:824] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:21.738229 11965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622: No bootstrap required, opened a new log
I20260806 22:44:21.749541 11965 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } }
I20260806 22:44:21.749977 11965 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:21.750092 11965 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a12a9884ba4d08aaadee5f4b0c7622, State: Initialized, Role: FOLLOWER
I20260806 22:44:21.750777 11965 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } }
I20260806 22:44:21.751786 12136 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } } }
I20260806 22:44:21.752288 12136 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:44:21.753125 11965 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 22:44:21.761348 12005 tablet.cc:2369] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:44:21.777962 11854 master_runner.cc:263] No leader master found from master a1a12a9884ba4d08aaadee5f4b0c7622
I20260806 22:44:21.778124 11854 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 22:44:21.781840 12151 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 22:44:21.782099 12151 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 22:44:21.967734 11890 tablet.cc:2369] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:44:22.033552 12136 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:44:22.033958 12136 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } }
I20260806 22:44:22.035810 12136 leader_election.cc:288] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8442e7642534939ab2d41267936e50f (127.23.100.190:40265), 1c1fca4d95a84ccb952ec24117c9a12e (127.23.100.188:46195)
I20260806 22:44:22.037379 12031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" is_pre_election: true
I20260806 22:44:22.038484 12031 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a12a9884ba4d08aaadee5f4b0c7622 in term 0.
I20260806 22:44:22.038283 11814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8442e7642534939ab2d41267936e50f" is_pre_election: true
I20260806 22:44:22.038923 11814 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a12a9884ba4d08aaadee5f4b0c7622 in term 0.
I20260806 22:44:22.039403 11885 leader_election.cc:302] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c1fca4d95a84ccb952ec24117c9a12e, a1a12a9884ba4d08aaadee5f4b0c7622; no voters:
I20260806 22:44:22.040230 12136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:44:22.040392 12136 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:44:22.040513 12136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:22.044747 12136 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } }
I20260806 22:44:22.046134 11814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8442e7642534939ab2d41267936e50f"
I20260806 22:44:22.046388 11814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:22.050220 11814 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a12a9884ba4d08aaadee5f4b0c7622 in term 1.
I20260806 22:44:22.050853 11883 leader_election.cc:302] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1a12a9884ba4d08aaadee5f4b0c7622, e8442e7642534939ab2d41267936e50f; no voters:
I20260806 22:44:22.061666 12031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c1fca4d95a84ccb952ec24117c9a12e"
I20260806 22:44:22.061987 12031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:22.066241 12031 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a12a9884ba4d08aaadee5f4b0c7622 in term 1.
I20260806 22:44:22.067276 12136 leader_election.cc:288] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [CANDIDATE]: Term 1 election: Requested vote from peers e8442e7642534939ab2d41267936e50f (127.23.100.190:40265), 1c1fca4d95a84ccb952ec24117c9a12e (127.23.100.188:46195)
I20260806 22:44:22.067754 12136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:44:22.068197 12136 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [term 1 LEADER]: Becoming Leader. State: Replica: a1a12a9884ba4d08aaadee5f4b0c7622, State: Running, Role: LEADER
I20260806 22:44:22.069101 12136 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } }
I20260806 22:44:22.080498 12155 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1a12a9884ba4d08aaadee5f4b0c7622. Latest consensus state: current_term: 1 leader_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } } }
I20260806 22:44:22.081013 12155 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:22.092005 12157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:44:22.094632 12157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:44:22.123575 12031 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [term 1 FOLLOWER]: Refusing update from remote peer a1a12a9884ba4d08aaadee5f4b0c7622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:44:22.124619 12155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:44:22.132431 11814 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [term 1 FOLLOWER]: Refusing update from remote peer a1a12a9884ba4d08aaadee5f4b0c7622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:44:22.133296 12155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:44:22.153254 12099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1a12a9884ba4d08aaadee5f4b0c7622. Latest consensus state: current_term: 1 leader_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } } }
I20260806 22:44:22.153647 12099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:44:22.160807 12160 mvcc.cc:205] Tried to move back new op lower bound from 7315686449644756992 to 7315686449447182336. Current Snapshot: MvccSnapshot[applied={T|T < 7315686449644756992}]
I20260806 22:44:22.162300 12119 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1a12a9884ba4d08aaadee5f4b0c7622. Latest consensus state: current_term: 1 leader_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } } }
I20260806 22:44:22.162616 12119 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:44:22.165005 12099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } } }
I20260806 22:44:22.165288 12099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:44:22.166433 12136 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } } }
I20260806 22:44:22.166739 12136 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:22.168076 12157 catalog_manager.cc:1347] Generated new cluster ID: d5c3727d43eb4730b9d64f95ea4fe47e
I20260806 22:44:22.168182 12157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:44:22.190647 12155 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } } }
I20260806 22:44:22.191004 12155 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:22.193634 12162 mvcc.cc:205] Tried to move back new op lower bound from 7315686449644756992 to 7315686449447182336. Current Snapshot: MvccSnapshot[applied={T|T < 7315686449644756992}]
I20260806 22:44:22.195514 12119 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8442e7642534939ab2d41267936e50f" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 40265 } } peers { permanent_uuid: "a1a12a9884ba4d08aaadee5f4b0c7622" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 35863 } } peers { permanent_uuid: "1c1fca4d95a84ccb952ec24117c9a12e" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 46195 } } }
I20260806 22:44:22.195783 12119 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:44:22.207757 12157 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:44:22.209865 12157 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:44:22.234997 12157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1a12a9884ba4d08aaadee5f4b0c7622: Generated new TSK 0
I20260806 22:44:22.236024 12157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:44:22.665341 12115 catalog_manager.cc:1259] Loaded cluster ID: d5c3727d43eb4730b9d64f95ea4fe47e
I20260806 22:44:22.665481 12115 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e: loading cluster ID for follower catalog manager: success
I20260806 22:44:22.667966 12115 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e: acquiring CA information for follower catalog manager: success
I20260806 22:44:22.669824 12115 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1c1fca4d95a84ccb952ec24117c9a12e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 22:44:22.687386 12131 catalog_manager.cc:1259] Loaded cluster ID: d5c3727d43eb4730b9d64f95ea4fe47e
I20260806 22:44:22.687525 12131 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f: loading cluster ID for follower catalog manager: success
I20260806 22:44:22.690044 12131 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f: acquiring CA information for follower catalog manager: success
I20260806 22:44:22.691826 12131 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e8442e7642534939ab2d41267936e50f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 22:44:22.831779 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 11757
I20260806 22:44:22.833132 11837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:22.945070 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 11757
I20260806 22:44:22.975837 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 11854
I20260806 22:44:22.977025 11956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:23.089469 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 11854
I20260806 22:44:23.115151 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 11974
I20260806 22:44:23.116290 12061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:23.224728 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 11974
2026-08-06T22:44:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4838 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:44:23 dist-test-slave-zr1t krb5kdc[12185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:44:23 dist-test-slave-zr1t krb5kdc[12185](info): set up 2 sockets
Aug 06 22:44:23 dist-test-slave-zr1t krb5kdc[12185](info): commencing operation
krb5kdc: starting...
W20260806 22:44:24.576478 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.303s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:44:24 dist-test-slave-zr1t krb5kdc[12185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:44:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:44:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:44:24.885174 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:34431
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:40369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:34431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:25.235481 12200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:25.235791 12200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:25.235960 12200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:25.243359 12200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:44:25.243490 12200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:25.243571 12200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:25.243687 12200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:44:25.243775 12200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:44:25.254024 12200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:40369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:34431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:34431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:25.255411 12200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:25.256788 12200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:25.269045 12208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:25.269021 12209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:25.269127 12212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:25.269964 12200 server_base.cc:1034] running on GCE node
I20260806 22:44:25.270839 12200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:25.272411 12200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:25.273656 12200 hybrid_clock.cc:648] HybridClock initialized: now 1786056265273581 us; error 55 us; skew 500 ppm
I20260806 22:44:25.274106 12200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:25 dist-test-slave-zr1t krb5kdc[12185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056265, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:25.277405 12200 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:44:25.278995 12200 webserver.cc:460] Webserver started at http://127.23.100.190:35779/ using document root <none> and password file <none>
I20260806 22:44:25.279662 12200 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:25.279764 12200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:25.280030 12200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:25.283078 12200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "0f8075071bd24d60b4ff0104ced2b977"
format_stamp: "Formatted at 2026-08-06 22:44:25 on dist-test-slave-zr1t"
server_key: "00e413072a5178b7b588350f5099d49d"
server_key_iv: "2600f8c8abb791271c73a598f998d6d8"
server_key_version: "encryptionkey@0"
I20260806 22:44:25.283798 12200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "0f8075071bd24d60b4ff0104ced2b977"
format_stamp: "Formatted at 2026-08-06 22:44:25 on dist-test-slave-zr1t"
server_key: "00e413072a5178b7b588350f5099d49d"
server_key_iv: "2600f8c8abb791271c73a598f998d6d8"
server_key_version: "encryptionkey@0"
I20260806 22:44:25.289158 12200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 22:44:25.292760 12226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:25.294209 12200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:44:25.294430 12200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "0f8075071bd24d60b4ff0104ced2b977"
format_stamp: "Formatted at 2026-08-06 22:44:25 on dist-test-slave-zr1t"
server_key: "00e413072a5178b7b588350f5099d49d"
server_key_iv: "2600f8c8abb791271c73a598f998d6d8"
server_key_version: "encryptionkey@0"
I20260806 22:44:25.294656 12200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:25.343744 12200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:25.348734 12200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:25.349067 12200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:25.369390 12200 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:34431
I20260806 22:44:25.369441 12294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:34431 every 8 connection(s)
I20260806 22:44:25.370965 12200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:44:25.375065 12295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:25.376354 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 12200
I20260806 22:44:25.376624 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:44:25.377559 23954 external_mini_cluster.cc:1428] Setting key 2ace392d007b529d9fa21f257ab3feb7
I20260806 22:44:25.391350 12295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977: Bootstrap starting.
Aug 06 22:44:25 dist-test-slave-zr1t krb5kdc[12185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:25.395713 12295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:25.396988 12295 log.cc:824] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:25.400374 12295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977: No bootstrap required, opened a new log
I20260806 22:44:25.404646 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:25.379862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.23.100.190:34431)
0806 22:44:25.380873 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:25.380886 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:25.380939 (+ 53us) server_negotiation.cc:407] Connection header received
0806 22:44:25.382920 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:25.382950 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:25.383470 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:25.383878 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:25.386060 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:25.387522 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:25.389620 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:25.390376 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:25.390523 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:25.396070 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:25.396085 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:25.396113 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:25.396163 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:25.399889 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:25.400745 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:25.400754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:25.400772 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:25.400973 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:25.401416 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:25.401430 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:25.401433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:25.402062 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:25.402411 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:25.403018 (+ 607us) server_negotiation.cc:299] Negotiation successful
0806 22:44:25.403532 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":467,"threads_started":1}
I20260806 22:44:25.409118 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:44:25.409613 12295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8075071bd24d60b4ff0104ced2b977" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34431 } }
I20260806 22:44:25.410048 12295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:25.410168 12295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8075071bd24d60b4ff0104ced2b977, State: Initialized, Role: FOLLOWER
I20260806 22:44:25.410844 12295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8075071bd24d60b4ff0104ced2b977" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34431 } }
I20260806 22:44:25.411132 12295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:44:25.411293 12295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:44:25.411494 12295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:25.414354 12295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8075071bd24d60b4ff0104ced2b977" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34431 } }
I20260806 22:44:25.414942 12295 leader_election.cc:302] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f8075071bd24d60b4ff0104ced2b977; no voters:
I20260806 22:44:25.415750 12295 leader_election.cc:288] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:44:25.415944 12300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:44:25.416764 12300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [term 1 LEADER]: Becoming Leader. State: Replica: 0f8075071bd24d60b4ff0104ced2b977, State: Running, Role: LEADER
I20260806 22:44:25.417506 12300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8075071bd24d60b4ff0104ced2b977" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34431 } }
I20260806 22:44:25.418318 12295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:44:25.421077 12302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f8075071bd24d60b4ff0104ced2b977. Latest consensus state: current_term: 1 leader_uuid: "0f8075071bd24d60b4ff0104ced2b977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8075071bd24d60b4ff0104ced2b977" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34431 } } }
I20260806 22:44:25.421577 12302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:25.421271 12301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f8075071bd24d60b4ff0104ced2b977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8075071bd24d60b4ff0104ced2b977" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34431 } } }
I20260806 22:44:25.421748 12301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:25.423756 12308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:44:25.428829 12308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:44:25.438711 12308 catalog_manager.cc:1347] Generated new cluster ID: c31acbc406a747ecba42352b9da7715c
I20260806 22:44:25.438853 12308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:44:25.455389 12308 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:44:25.457394 12308 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:44:25.466967 12308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f8075071bd24d60b4ff0104ced2b977: Generated new TSK 0
I20260806 22:44:25.467841 12308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:44:25.522778 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 12200
I20260806 22:44:25.524179 12288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:25.626293 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 12200
2026-08-06T22:44:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2396 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:44:25 dist-test-slave-zr1t krb5kdc[12334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:44:25 dist-test-slave-zr1t krb5kdc[12334](info): set up 2 sockets
Aug 06 22:44:25 dist-test-slave-zr1t krb5kdc[12334](info): commencing operation
krb5kdc: starting...
W20260806 22:44:26.995741 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.002s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:44:27 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:44:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:44:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:44:27.309242 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:36257
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:36257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:27.661804 12349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:27.662097 12349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:27.662274 12349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:27.669785 12349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:44:27.669893 12349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:27.670017 12349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:27.670092 12349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:44:27.670156 12349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:44:27.680151 12349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:36257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:27.681620 12349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:27.682966 12349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:27.695047 12359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:27.695048 12362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:27.695369 12360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:27.696060 12349 server_base.cc:1034] running on GCE node
I20260806 22:44:27.696834 12349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:27.698370 12349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:27.699596 12349 hybrid_clock.cc:648] HybridClock initialized: now 1786056267699543 us; error 39 us; skew 500 ppm
I20260806 22:44:27.699963 12349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:27 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056267, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:27.708148 12349 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:44:27.709941 12349 webserver.cc:460] Webserver started at http://127.23.100.190:41497/ using document root <none> and password file <none>
I20260806 22:44:27.710673 12349 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:27.710829 12349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:27.711145 12349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:27.714493 12349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "1c09fa0a3e8043baa7b70a074235a6f8"
format_stamp: "Formatted at 2026-08-06 22:44:27 on dist-test-slave-zr1t"
server_key: "dcdebb2abf2841d4506148ae51eeed90"
server_key_iv: "1e9e891f8064d3621fdd983d560e667f"
server_key_version: "encryptionkey@0"
I20260806 22:44:27.715401 12349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "1c09fa0a3e8043baa7b70a074235a6f8"
format_stamp: "Formatted at 2026-08-06 22:44:27 on dist-test-slave-zr1t"
server_key: "dcdebb2abf2841d4506148ae51eeed90"
server_key_iv: "1e9e891f8064d3621fdd983d560e667f"
server_key_version: "encryptionkey@0"
I20260806 22:44:27.720779 12349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 22:44:27.724552 12374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:27.726099 12349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:44:27.726310 12349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "1c09fa0a3e8043baa7b70a074235a6f8"
format_stamp: "Formatted at 2026-08-06 22:44:27 on dist-test-slave-zr1t"
server_key: "dcdebb2abf2841d4506148ae51eeed90"
server_key_iv: "1e9e891f8064d3621fdd983d560e667f"
server_key_version: "encryptionkey@0"
I20260806 22:44:27.726538 12349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:27.743752 12349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:27.748486 12349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:27.748836 12349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:27.769589 12349 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:36257
I20260806 22:44:27.769677 12457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:36257 every 8 connection(s)
I20260806 22:44:27.771044 12349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:44:27.772161 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 12349
I20260806 22:44:27.772439 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:44:27.773348 23954 external_mini_cluster.cc:1428] Setting key f6f4910095026bfe7a4b62847bc4c7ba
I20260806 22:44:27.776157 12458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 22:44:27 dist-test-slave-zr1t krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:27.790120 12458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Bootstrap starting.
I20260806 22:44:27.794483 12458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:27.795867 12458 log.cc:824] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:27.799769 12458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: No bootstrap required, opened a new log
I20260806 22:44:27.800486 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:27.775599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.23.100.190:36257)
0806 22:44:27.776472 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:27.776484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:27.776527 (+ 43us) server_negotiation.cc:407] Connection header received
0806 22:44:27.778248 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:27.778272 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:27.778665 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:27.779012 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:27.780933 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:27.782680 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:27.784607 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:27.785294 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:27.785474 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:27.791002 (+ 5528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:27.791015 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:27.791034 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:27.791084 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:27.795360 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:27.796200 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:27.796209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:27.796217 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:27.796472 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:27.797096 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:27.797100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:27.797102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:27.797846 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:27.798169 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:27.798633 (+ 464us) server_negotiation.cc:299] Negotiation successful
0806 22:44:27.799123 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":397,"threads_started":1}
I20260806 22:44:27.808382 12458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:27.808761 12458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:27.808832 12458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c09fa0a3e8043baa7b70a074235a6f8, State: Initialized, Role: FOLLOWER
I20260806 22:44:27.809391 12458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:27.809686 12458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:44:27.809837 12458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:44:27.810065 12458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:27.812374 12458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:27.812884 12458 leader_election.cc:302] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c09fa0a3e8043baa7b70a074235a6f8; no voters:
I20260806 22:44:27.813566 12458 leader_election.cc:288] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:44:27.813741 12468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:44:27.814474 12468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 1 LEADER]: Becoming Leader. State: Replica: 1c09fa0a3e8043baa7b70a074235a6f8, State: Running, Role: LEADER
I20260806 22:44:27.815086 12468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:27.815677 12458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:44:27.817965 12470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c09fa0a3e8043baa7b70a074235a6f8. Latest consensus state: current_term: 1 leader_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } } }
I20260806 22:44:27.818372 12470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:27.819154 12469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } } }
I20260806 22:44:27.819485 12469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:27.821099 12473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:44:27.826423 12473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:44:27.835556 12473 catalog_manager.cc:1347] Generated new cluster ID: 0038493f127c4991abdae7edf798f59b
I20260806 22:44:27.835709 12473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:44:27.850001 12473 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:44:27.851336 12473 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:44:27.860011 12473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Generated new TSK 0
I20260806 22:44:27.860868 12473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:44:27.939255 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:36257
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:28.295167 12508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:28.295486 12508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:28.295588 12508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:28.303506 12508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:28.303674 12508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:28.303849 12508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:44:28.314193 12508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:28.315624 12508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:28.317144 12508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:28.331039 12515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:28.331113 12521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:28.331033 12516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:28.331741 12508 server_base.cc:1034] running on GCE node
I20260806 22:44:28.332495 12508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:28.333452 12508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:28.334739 12508 hybrid_clock.cc:648] HybridClock initialized: now 1786056268334692 us; error 36 us; skew 500 ppm
I20260806 22:44:28.335042 12508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:28 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056268, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:28.338420 12508 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:44:28.340162 12508 webserver.cc:460] Webserver started at http://127.23.100.129:34449/ using document root <none> and password file <none>
I20260806 22:44:28.340951 12508 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:28.341106 12508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:28.341452 12508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:28.344679 12508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "54fdc651763245718a8b625e727046a5"
format_stamp: "Formatted at 2026-08-06 22:44:28 on dist-test-slave-zr1t"
server_key: "e09aedc9b2f847154092bd502b780c18"
server_key_iv: "1824a47c3799250bd6156c8a49f94ef3"
server_key_version: "encryptionkey@0"
I20260806 22:44:28.345587 12508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "54fdc651763245718a8b625e727046a5"
format_stamp: "Formatted at 2026-08-06 22:44:28 on dist-test-slave-zr1t"
server_key: "e09aedc9b2f847154092bd502b780c18"
server_key_iv: "1824a47c3799250bd6156c8a49f94ef3"
server_key_version: "encryptionkey@0"
I20260806 22:44:28.350968 12508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 22:44:28.354740 12540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:28.356083 12508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:44:28.356287 12508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "54fdc651763245718a8b625e727046a5"
format_stamp: "Formatted at 2026-08-06 22:44:28 on dist-test-slave-zr1t"
server_key: "e09aedc9b2f847154092bd502b780c18"
server_key_iv: "1824a47c3799250bd6156c8a49f94ef3"
server_key_version: "encryptionkey@0"
I20260806 22:44:28.356500 12508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:28.370985 12508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:28.375837 12508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:28.376159 12508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:28.377206 12508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:28.378647 12508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:28.378721 12508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:28.378881 12508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:28.378947 12508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:28.413319 12508 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:43837
I20260806 22:44:28.413367 12709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:43837 every 8 connection(s)
I20260806 22:44:28.414654 12508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:44:28.415594 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 12508
I20260806 22:44:28.415840 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:44:28.416651 23954 external_mini_cluster.cc:1428] Setting key cab0c7e398d26d3f6ab8977a01522632
Aug 06 22:44:28 dist-test-slave-zr1t krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056268, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:28.445384 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:28.417791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:36933 (local address 127.23.100.190:36257)
0806 22:44:28.425933 (+ 8142us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:28.425937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:28.426521 (+ 584us) server_negotiation.cc:407] Connection header received
0806 22:44:28.428416 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:28.428422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:28.428522 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:28.428644 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:28.430332 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:28.431134 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:28.432729 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:28.433141 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:28.433295 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:28.437576 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:28.437586 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:28.437595 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:28.437643 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:28.441377 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:28.442634 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:28.442643 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:28.442649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:28.442836 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:28.443310 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:28.443318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:28.443324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:28.443731 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:28.443893 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:28.444612 (+ 719us) server_negotiation.cc:299] Negotiation successful
0806 22:44:28.444878 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8035,"thread_start_us":7913,"threads_started":1}
I20260806 22:44:28.446051 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:28.418349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:36933)
0806 22:44:28.426349 (+ 8000us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:28.426393 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:28.428119 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:28.428890 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:28.428898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:28.429438 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:28.430165 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:28.430185 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:28.431296 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:28.431301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:28.432573 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:28.432586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:28.433415 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:28.433422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:28.433727 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:28.434480 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:28.434503 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:28.437281 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:28.441645 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:28.441658 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:28.441676 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:28.442416 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:28.442960 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:28.442966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:28.442969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:28.443166 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:28.443851 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:28.443857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:28.444192 (+ 335us) client_negotiation.cc:769] Sending connection context
0806 22:44:28.444413 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 22:44:28.444855 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7734,"thread_start_us":7556,"threads_started":1}
I20260806 22:44:28.447731 12710 heartbeater.cc:347] Connected to a master server at 127.23.100.190:36257
I20260806 22:44:28.448071 12710 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:28.448927 12710 heartbeater.cc:511] Master 127.23.100.190:36257 requested a full tablet report, sending...
I20260806 22:44:28.451151 12400 ts_manager.cc:194] Registered new tserver with Master: 54fdc651763245718a8b625e727046a5 (127.23.100.129:43837)
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
I20260806 22:44:28.452689 12400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:36933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:44:28.492287 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:36257
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:28.855657 12717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:28.856019 12717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:28.856199 12717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:28.863791 12717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:28.863962 12717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:28.864143 12717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:44:28.874053 12717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:28.875476 12717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:28.876849 12717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:28.890569 12726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:28.890489 12727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:28.890993 12729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:28.891633 12717 server_base.cc:1034] running on GCE node
I20260806 22:44:28.892270 12717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:28.893255 12717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:28.894510 12717 hybrid_clock.cc:648] HybridClock initialized: now 1786056268894454 us; error 52 us; skew 500 ppm
I20260806 22:44:28.894867 12717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:28 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056268, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:28.898433 12717 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:44:28.900053 12717 webserver.cc:460] Webserver started at http://127.23.100.130:36927/ using document root <none> and password file <none>
I20260806 22:44:28.900768 12717 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:28.900928 12717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:28.901248 12717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:28.904318 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "4537b6cc502a4ff3a548c7a7125c9734"
format_stamp: "Formatted at 2026-08-06 22:44:28 on dist-test-slave-zr1t"
server_key: "5a1d2e0155f055c0c33821934c203985"
server_key_iv: "57180a39fcbc5a97a8d89d8b0f42cab7"
server_key_version: "encryptionkey@0"
I20260806 22:44:28.905138 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "4537b6cc502a4ff3a548c7a7125c9734"
format_stamp: "Formatted at 2026-08-06 22:44:28 on dist-test-slave-zr1t"
server_key: "5a1d2e0155f055c0c33821934c203985"
server_key_iv: "57180a39fcbc5a97a8d89d8b0f42cab7"
server_key_version: "encryptionkey@0"
I20260806 22:44:28.910601 12717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260806 22:44:28.914471 12740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:28.916046 12717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:44:28.916260 12717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "4537b6cc502a4ff3a548c7a7125c9734"
format_stamp: "Formatted at 2026-08-06 22:44:28 on dist-test-slave-zr1t"
server_key: "5a1d2e0155f055c0c33821934c203985"
server_key_iv: "57180a39fcbc5a97a8d89d8b0f42cab7"
server_key_version: "encryptionkey@0"
I20260806 22:44:28.916491 12717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:28.938366 12717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:28.943110 12717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:28.943421 12717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:28.944432 12717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:28.945865 12717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:28.945974 12717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:28.946121 12717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:28.946193 12717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:28.982117 12717 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:36179
I20260806 22:44:28.982180 12913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:36179 every 8 connection(s)
I20260806 22:44:28.983505 12717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:44:28.984604 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 12717
I20260806 22:44:28.984820 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:44:28.985435 23954 external_mini_cluster.cc:1428] Setting key 7037042b7fda7feae9120bb9660a13af
Aug 06 22:44:29 dist-test-slave-zr1t krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056268, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:29.011473 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:28.986655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:50633 (local address 127.23.100.190:36257)
0806 22:44:28.987148 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:28.987152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:28.988771 (+ 1619us) server_negotiation.cc:407] Connection header received
0806 22:44:28.992116 (+ 3345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:28.992120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:28.992192 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:28.992295 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:28.994732 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:28.995510 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:28.998350 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:28.998834 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:28.998955 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.004408 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:29.004413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:29.004417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:29.004453 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:29.007396 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:29.008597 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:29.008602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:29.008604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:29.008749 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:29.009346 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:29.009350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:29.009352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:29.009722 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:29.009848 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:29.010973 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0806 22:44:29.011161 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":282,"threads_started":1}
I20260806 22:44:29.013223 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:28.987468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:50633)
0806 22:44:28.988560 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:28.988615 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:28.991733 (+ 3118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:28.992602 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:28.992617 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:28.993561 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:28.994503 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:28.994537 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:28.995695 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:28.995704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:28.998154 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:28.998174 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:28.999113 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:28.999124 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:28.999517 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.000393 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:29.000433 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:29.004088 (+ 3655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:29.007572 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:29.007585 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:29.007611 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:29.008405 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:29.008887 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:29.008895 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:29.008902 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:29.009180 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:29.009861 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:29.009872 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:29.010491 (+ 619us) client_negotiation.cc:769] Sending connection context
0806 22:44:29.010849 (+ 358us) client_negotiation.cc:240] Negotiation successful
0806 22:44:29.011554 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":436,"threads_started":1}
I20260806 22:44:29.015024 12915 heartbeater.cc:347] Connected to a master server at 127.23.100.190:36257
I20260806 22:44:29.015388 12915 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:29.016254 12915 heartbeater.cc:511] Master 127.23.100.190:36257 requested a full tablet report, sending...
I20260806 22:44:29.018025 12400 ts_manager.cc:194] Registered new tserver with Master: 4537b6cc502a4ff3a548c7a7125c9734 (127.23.100.130:36179)
I20260806 22:44:29.019603 12400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:50633
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:44:29.066366 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:36257
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:29.413678 12923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:29.413980 12923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:29.414135 12923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:29.421936 12923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:29.422047 12923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:29.422308 12923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:44:29.432268 12923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:29.433634 12923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:29.434960 12923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:29.448467 12932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:29.448520 12934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:29.448498 12931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:29.449720 12923 server_base.cc:1034] running on GCE node
I20260806 22:44:29.450290 12923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:29.451201 12923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:29.452399 12923 hybrid_clock.cc:648] HybridClock initialized: now 1786056269452343 us; error 71 us; skew 500 ppm
I20260806 22:44:29.452673 12923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:29 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056269, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:29.455834 12710 heartbeater.cc:503] Master 127.23.100.190:36257 was elected leader, sending a full tablet report...
I20260806 22:44:29.456125 12923 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:44:29.457918 12923 webserver.cc:460] Webserver started at http://127.23.100.131:35819/ using document root <none> and password file <none>
I20260806 22:44:29.458670 12923 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:29.458848 12923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:29.459172 12923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:29.462354 12923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "04cc6269cc1445528576a42ce5c1ef66"
format_stamp: "Formatted at 2026-08-06 22:44:29 on dist-test-slave-zr1t"
server_key: "39a0fdcae37c0e6dd5ef14b554148cb0"
server_key_iv: "26f6020af0e5a42acccf20e189a85833"
server_key_version: "encryptionkey@0"
I20260806 22:44:29.463201 12923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "04cc6269cc1445528576a42ce5c1ef66"
format_stamp: "Formatted at 2026-08-06 22:44:29 on dist-test-slave-zr1t"
server_key: "39a0fdcae37c0e6dd5ef14b554148cb0"
server_key_iv: "26f6020af0e5a42acccf20e189a85833"
server_key_version: "encryptionkey@0"
I20260806 22:44:29.468663 12923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:44:29.472563 12946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:29.474119 12923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 22:44:29.474310 12923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "04cc6269cc1445528576a42ce5c1ef66"
format_stamp: "Formatted at 2026-08-06 22:44:29 on dist-test-slave-zr1t"
server_key: "39a0fdcae37c0e6dd5ef14b554148cb0"
server_key_iv: "26f6020af0e5a42acccf20e189a85833"
server_key_version: "encryptionkey@0"
I20260806 22:44:29.474572 12923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:29.506524 12923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:29.511123 12923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:29.511456 12923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:29.512564 12923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:29.514132 12923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:29.514243 12923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:29.514401 12923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:29.514485 12923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:29.551183 12923 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:42519
I20260806 22:44:29.551246 13106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:42519 every 8 connection(s)
I20260806 22:44:29.552553 12923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:44:29.559201 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 12923
I20260806 22:44:29.559496 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:44:29.560056 23954 external_mini_cluster.cc:1428] Setting key 138ad7e0c9562447ffc53e9f7e3ea69a
Aug 06 22:44:29 dist-test-slave-zr1t krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056269, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:29.573807 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.555173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:39425 (local address 127.23.100.190:36257)
0806 22:44:29.555663 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:29.555670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:29.556613 (+ 943us) server_negotiation.cc:407] Connection header received
0806 22:44:29.558510 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:29.558514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:29.558585 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:29.558685 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:29.560478 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.561688 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.563545 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.563958 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.564097 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.567922 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:29.567933 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:29.567936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:29.567972 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:29.570660 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:29.571417 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:29.571425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:29.571430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:29.571565 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:29.572000 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:29.572006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:29.572009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:29.572283 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:29.572465 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:29.573143 (+ 678us) server_negotiation.cc:299] Negotiation successful
0806 22:44:29.573365 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":216,"threads_started":1}
I20260806 22:44:29.575039 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.555756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:39425)
0806 22:44:29.556453 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:29.556493 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:29.558232 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:29.558899 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:29.558907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:29.559524 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:29.560275 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.560297 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.561869 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.561873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.563402 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.563415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.564197 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.564204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.564439 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.565172 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:29.565196 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:29.567688 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:29.570822 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:29.570831 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:29.570850 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:29.571293 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:29.571679 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:29.571684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:29.571688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:29.571880 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:29.572415 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:29.572425 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:29.572785 (+ 360us) client_negotiation.cc:769] Sending connection context
0806 22:44:29.573065 (+ 280us) client_negotiation.cc:240] Negotiation successful
0806 22:44:29.573635 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":287,"threads_started":1}
I20260806 22:44:29.576455 13107 heartbeater.cc:347] Connected to a master server at 127.23.100.190:36257
I20260806 22:44:29.576778 13107 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:29.577592 13107 heartbeater.cc:511] Master 127.23.100.190:36257 requested a full tablet report, sending...
I20260806 22:44:29.579090 12400 ts_manager.cc:194] Registered new tserver with Master: 04cc6269cc1445528576a42ce5c1ef66 (127.23.100.131:42519)
I20260806 22:44:29.580019 12400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:39425
I20260806 22:44:29.588965 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:44:29.607045 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.593962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.23.100.190:36257)
0806 22:44:29.594178 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:29.594183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:29.594650 (+ 467us) server_negotiation.cc:407] Connection header received
0806 22:44:29.594837 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:29.594841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:29.594909 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:29.595002 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:29.596445 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.597209 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.598691 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.599115 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.599227 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.601852 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:29.601857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:29.601860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:29.601891 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:29.604331 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:29.604978 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:29.604984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:29.604987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:29.605120 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:29.605669 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:29.605673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:29.605675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:29.606017 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:29.606230 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:29.606612 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 22:44:29.606798 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 22:44:29.611296 12400 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:44:29.618495 12400 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:44:29.642095 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.632154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:36179 (local address 127.0.0.1:55028)
0806 22:44:29.632888 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:29.632940 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:29.633124 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:29.634137 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:29.634146 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:29.634185 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:29.634729 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.634744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.636940 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.636945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.639512 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.639527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.640953 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.640959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.641143 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.641193 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 22:44:29.641347 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 22:44:29.641580 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":114,"thread_start_us":751,"threads_started":1}
I20260806 22:44:29.642187 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.632752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55028 (local address 127.23.100.130:36179)
0806 22:44:29.633444 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:29.633452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:29.633483 (+ 31us) server_negotiation.cc:407] Connection header received
0806 22:44:29.633610 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:29.633618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:29.633878 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:29.634926 (+ 1048us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:29.635180 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.636792 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.639704 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.640721 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.640932 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.641109 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:29.641658 (+ 549us) server_negotiation.cc:299] Negotiation successful
0806 22:44:29.641833 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":763,"threads_started":1}
I20260806 22:44:29.643311 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.632192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:42519 (local address 127.0.0.1:41426)
0806 22:44:29.633724 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:29.633747 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:29.633871 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:29.634745 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:29.634750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:29.634773 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:29.635218 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.635234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.637254 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.637262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.639503 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.639517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.641393 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.641399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.642608 (+ 1209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.642655 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 22:44:29.642758 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 22:44:29.642889 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1415,"mutex_wait_us":777,"thread_start_us":1231,"threads_started":1}
I20260806 22:44:29.643352 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.632963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41426 (local address 127.23.100.131:42519)
0806 22:44:29.634160 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:29.634170 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:29.634205 (+ 35us) server_negotiation.cc:407] Connection header received
0806 22:44:29.634304 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:29.634314 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:29.634532 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:29.635411 (+ 879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:29.635667 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.637126 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.639705 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.641174 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.641355 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.641554 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:29.642836 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0806 22:44:29.643023 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"spinlock_wait_cycles":20864,"thread_start_us":1000,"threads_started":1}
I20260806 22:44:29.645598 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.633796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43837 (local address 127.0.0.1:41066)
0806 22:44:29.634615 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:29.634638 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:29.634745 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:29.637313 (+ 2568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:29.637319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:29.637349 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:29.637943 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.637964 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.639901 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.639908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.641977 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.641993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.644736 (+ 2743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.644747 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.644941 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.644977 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 22:44:29.645077 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 22:44:29.645198 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":1,"thread_start_us":827,"threads_started":1}
I20260806 22:44:29.645686 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.634411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.23.100.129:43837)
0806 22:44:29.636757 (+ 2346us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:29.636763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:29.636785 (+ 22us) server_negotiation.cc:407] Connection header received
0806 22:44:29.636887 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:29.636895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:29.637119 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:29.637307 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:29.638188 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.639756 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.642180 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.644502 (+ 2322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.644729 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.644897 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:29.645140 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 22:44:29.645316 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2205,"thread_start_us":2065,"threads_started":1}
I20260806 22:44:29.646793 12813 tablet_service.cc:1467] Processing CreateTablet for tablet be19b3d8910b41bd8c0c9b1aac96890e (DEFAULT_TABLE table=test-table [id=eb781bfe10cd41fe8eaa02974d4efe5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:29.647121 13010 tablet_service.cc:1467] Processing CreateTablet for tablet be19b3d8910b41bd8c0c9b1aac96890e (DEFAULT_TABLE table=test-table [id=eb781bfe10cd41fe8eaa02974d4efe5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:29.648419 12813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be19b3d8910b41bd8c0c9b1aac96890e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:29.648447 13010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be19b3d8910b41bd8c0c9b1aac96890e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:29.649482 12619 tablet_service.cc:1467] Processing CreateTablet for tablet be19b3d8910b41bd8c0c9b1aac96890e (DEFAULT_TABLE table=test-table [id=eb781bfe10cd41fe8eaa02974d4efe5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:29.651055 12619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be19b3d8910b41bd8c0c9b1aac96890e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:29.660651 13137 tablet_bootstrap.cc:492] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66: Bootstrap starting.
I20260806 22:44:29.663373 13139 tablet_bootstrap.cc:492] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: Bootstrap starting.
I20260806 22:44:29.665153 13137 tablet_bootstrap.cc:654] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:29.665633 13140 tablet_bootstrap.cc:492] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Bootstrap starting.
I20260806 22:44:29.666766 13137 log.cc:824] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:29.667107 13139 tablet_bootstrap.cc:654] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:29.668205 13139 log.cc:824] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:29.668825 13140 tablet_bootstrap.cc:654] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:29.670208 13137 tablet_bootstrap.cc:492] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66: No bootstrap required, opened a new log
I20260806 22:44:29.670326 13140 log.cc:824] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:29.670497 13137 ts_tablet_manager.cc:1397] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260806 22:44:29.671823 13139 tablet_bootstrap.cc:492] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: No bootstrap required, opened a new log
I20260806 22:44:29.672096 13139 ts_tablet_manager.cc:1397] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 22:44:29.675622 13140 tablet_bootstrap.cc:492] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: No bootstrap required, opened a new log
I20260806 22:44:29.675894 13140 ts_tablet_manager.cc:1397] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260806 22:44:29.681694 13137 raft_consensus.cc:348] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:29.682142 13137 raft_consensus.cc:374] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:29.682272 13137 raft_consensus.cc:729] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04cc6269cc1445528576a42ce5c1ef66, State: Initialized, Role: FOLLOWER
I20260806 22:44:29.682693 13139 raft_consensus.cc:348] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:29.683117 13139 raft_consensus.cc:374] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:29.682925 13137 consensus_queue.cc:260] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:29.683214 13139 raft_consensus.cc:729] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4537b6cc502a4ff3a548c7a7125c9734, State: Initialized, Role: FOLLOWER
I20260806 22:44:29.684123 13107 heartbeater.cc:503] Master 127.23.100.190:36257 was elected leader, sending a full tablet report...
I20260806 22:44:29.684494 13140 raft_consensus.cc:348] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:29.684791 13137 ts_tablet_manager.cc:1428] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 22:44:29.684821 13140 raft_consensus.cc:374] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:29.684929 13140 raft_consensus.cc:729] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54fdc651763245718a8b625e727046a5, State: Initialized, Role: FOLLOWER
I20260806 22:44:29.685061 13139 consensus_queue.cc:260] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:29.685581 13140 consensus_queue.cc:260] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:29.686612 12915 heartbeater.cc:503] Master 127.23.100.190:36257 was elected leader, sending a full tablet report...
I20260806 22:44:29.687547 13139 ts_tablet_manager.cc:1428] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260806 22:44:29.689025 13140 ts_tablet_manager.cc:1428] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260806 22:44:29.707154 13151 raft_consensus.cc:482] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:44:29.707507 13151 raft_consensus.cc:504] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:29.709240 13151 leader_election.cc:288] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04cc6269cc1445528576a42ce5c1ef66 (127.23.100.131:42519), 54fdc651763245718a8b625e727046a5 (127.23.100.129:43837)
I20260806 22:44:29.716558 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.709829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:47665 (local address 127.23.100.131:42519)
0806 22:44:29.710082 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:29.710088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:29.710573 (+ 485us) server_negotiation.cc:407] Connection header received
0806 22:44:29.710938 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:29.710947 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:29.711053 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:29.711133 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:29.712071 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.712866 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.714726 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.715519 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.715713 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.715794 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:29.716135 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 22:44:29.716270 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 22:44:29.716621 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.709705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:42519 (local address 127.23.100.130:47665)
0806 22:44:29.710471 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:29.710493 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:29.710711 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:29.711393 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:29.711397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:29.711439 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:29.711920 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.711931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.712997 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.713004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.714565 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.714577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.715741 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.715747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.715973 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.715998 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 22:44:29.716107 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 22:44:29.716238 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":406,"threads_started":1}
I20260806 22:44:29.717267 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.710865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:40885 (local address 127.23.100.129:43837)
0806 22:44:29.711047 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:29.711051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:29.711996 (+ 945us) server_negotiation.cc:407] Connection header received
0806 22:44:29.712194 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:29.712198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:29.712269 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:29.712391 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:29.712918 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.713680 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.715368 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.716257 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.716455 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.716529 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:29.716914 (+ 385us) server_negotiation.cc:299] Negotiation successful
0806 22:44:29.717099 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 22:44:29.717935 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.710599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43837 (local address 127.23.100.130:40885)
0806 22:44:29.711886 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:29.711903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:29.712047 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:29.712418 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:29.712422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:29.712449 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:29.712772 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.712780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.713822 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.713827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.715209 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:29.715221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.716488 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.716494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:29.716704 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.716752 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 22:44:29.717477 (+ 725us) client_negotiation.cc:240] Negotiation successful
0806 22:44:29.717642 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"mutex_wait_us":4,"thread_start_us":991,"threads_started":1}
I20260806 22:44:29.717763 13046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be19b3d8910b41bd8c0c9b1aac96890e" candidate_uuid: "4537b6cc502a4ff3a548c7a7125c9734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04cc6269cc1445528576a42ce5c1ef66" is_pre_election: true
I20260806 22:44:29.718319 13046 raft_consensus.cc:2454] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4537b6cc502a4ff3a548c7a7125c9734 in term 0.
I20260806 22:44:29.719089 12743 leader_election.cc:302] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04cc6269cc1445528576a42ce5c1ef66, 4537b6cc502a4ff3a548c7a7125c9734; no voters:
I20260806 22:44:29.719094 12651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be19b3d8910b41bd8c0c9b1aac96890e" candidate_uuid: "4537b6cc502a4ff3a548c7a7125c9734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54fdc651763245718a8b625e727046a5" is_pre_election: true
I20260806 22:44:29.719558 13151 raft_consensus.cc:2781] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:44:29.719733 13151 raft_consensus.cc:482] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:44:29.719646 12651 raft_consensus.cc:2454] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4537b6cc502a4ff3a548c7a7125c9734 in term 0.
I20260806 22:44:29.719873 13151 raft_consensus.cc:3037] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:29.722697 13151 raft_consensus.cc:504] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:29.723351 13151 leader_election.cc:288] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [CANDIDATE]: Term 1 election: Requested vote from peers 04cc6269cc1445528576a42ce5c1ef66 (127.23.100.131:42519), 54fdc651763245718a8b625e727046a5 (127.23.100.129:43837)
I20260806 22:44:29.723933 13046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be19b3d8910b41bd8c0c9b1aac96890e" candidate_uuid: "4537b6cc502a4ff3a548c7a7125c9734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04cc6269cc1445528576a42ce5c1ef66"
I20260806 22:44:29.723978 12651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be19b3d8910b41bd8c0c9b1aac96890e" candidate_uuid: "4537b6cc502a4ff3a548c7a7125c9734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54fdc651763245718a8b625e727046a5"
I20260806 22:44:29.724172 12651 raft_consensus.cc:3037] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:29.724172 13046 raft_consensus.cc:3037] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:29.727574 13046 raft_consensus.cc:2454] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4537b6cc502a4ff3a548c7a7125c9734 in term 1.
I20260806 22:44:29.727591 12651 raft_consensus.cc:2454] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4537b6cc502a4ff3a548c7a7125c9734 in term 1.
I20260806 22:44:29.728086 12743 leader_election.cc:302] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04cc6269cc1445528576a42ce5c1ef66, 4537b6cc502a4ff3a548c7a7125c9734; no voters:
I20260806 22:44:29.728417 13151 raft_consensus.cc:2781] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:44:29.728900 13151 raft_consensus.cc:686] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 1 LEADER]: Becoming Leader. State: Replica: 4537b6cc502a4ff3a548c7a7125c9734, State: Running, Role: LEADER
I20260806 22:44:29.729406 13151 consensus_queue.cc:237] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:29.734843 12400 catalog_manager.cc:5495] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4537b6cc502a4ff3a548c7a7125c9734 (127.23.100.130). New cstate: current_term: 1 leader_uuid: "4537b6cc502a4ff3a548c7a7125c9734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } health_report { overall_health: UNKNOWN } } }
W20260806 22:44:29.735677 12916 tablet.cc:2369] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:44:29.768165 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.762571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.23.100.130:36179)
0806 22:44:29.762766 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:29.762771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:29.762790 (+ 19us) server_negotiation.cc:407] Connection header received
0806 22:44:29.762941 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:29.762945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:29.763035 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:29.763166 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:44:29.763526 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.764244 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.765900 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.766344 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.766449 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.767048 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:44:29.767586 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:44:29.767701 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:29.767860 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 22:44:29.767936 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 22:44:29.777164 13046 raft_consensus.cc:1261] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66 [term 1 FOLLOWER]: Refusing update from remote peer 4537b6cc502a4ff3a548c7a7125c9734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:44:29.777181 12651 raft_consensus.cc:1261] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 1 FOLLOWER]: Refusing update from remote peer 4537b6cc502a4ff3a548c7a7125c9734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:44:29.778313 13157 consensus_queue.cc:1035] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:44:29.778731 13151 consensus_queue.cc:1035] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:44:29.791500 13161 mvcc.cc:205] Tried to move back new op lower bound from 7315686480996073472 to 7315686480816726016. Current Snapshot: MvccSnapshot[applied={T|T < 7315686480996073472}]
I20260806 22:44:29.792383 13162 mvcc.cc:205] Tried to move back new op lower bound from 7315686480996073472 to 7315686480816726016. Current Snapshot: MvccSnapshot[applied={T|T < 7315686480996073472}]
I20260806 22:44:29.798727 13164 mvcc.cc:205] Tried to move back new op lower bound from 7315686480996073472 to 7315686480816726016. Current Snapshot: MvccSnapshot[applied={T|T < 7315686480996073472}]
I20260806 22:44:29.808113 12399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:44:29.808431 12399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:44:29.812287 12399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Sending DeleteTablet for 3 replicas of tablet be19b3d8910b41bd8c0c9b1aac96890e
I20260806 22:44:29.813591 12813 tablet_service.cc:1514] Processing DeleteTablet for tablet be19b3d8910b41bd8c0c9b1aac96890e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:29 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:55028
I20260806 22:44:29.813776 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 12508
I20260806 22:44:29.813748 13010 tablet_service.cc:1514] Processing DeleteTablet for tablet be19b3d8910b41bd8c0c9b1aac96890e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:29 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:41426
I20260806 22:44:29.813858 12619 tablet_service.cc:1514] Processing DeleteTablet for tablet be19b3d8910b41bd8c0c9b1aac96890e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:29 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:41066
I20260806 22:44:29.824546 13174 tablet_replica.cc:330] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: stopping tablet replica
I20260806 22:44:29.824872 13172 tablet_replica.cc:330] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: stopping tablet replica
I20260806 22:44:29.825440 13172 raft_consensus.cc:2229] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:44:29.826241 13174 raft_consensus.cc:2229] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:44:29.826329 13172 raft_consensus.cc:2258] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:29.826799 13174 raft_consensus.cc:2258] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 06 22:44:29 dist-test-slave-zr1t krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:44:29.829442 13172 ts_tablet_manager.cc:1905] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:29.831147 13173 tablet_replica.cc:330] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66: stopping tablet replica
I20260806 22:44:29.831759 13173 raft_consensus.cc:2229] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:44:29.832494 13173 raft_consensus.cc:2258] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:29.835225 13173 ts_tablet_manager.cc:1905] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:29.839299 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.814845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.23.100.129:43837)
0806 22:44:29.815090 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:29.815098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:29.815414 (+ 316us) server_negotiation.cc:407] Connection header received
0806 22:44:29.815746 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:29.815755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:29.815864 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:29.815980 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:29.818700 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.820275 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.823071 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.823801 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:29.823921 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:29.830528 (+ 6607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:29.830548 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:29.830567 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:29.830628 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:29.835239 (+ 4611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:29.836406 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:29.836417 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:29.836421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:29.836607 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:29.837135 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:29.837140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:29.837143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:29.837718 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:29.838026 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:29.838652 (+ 626us) server_negotiation.cc:299] Negotiation successful
0806 22:44:29.838923 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 22:44:29.839578 13172 ts_tablet_manager.cc:1918] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:29.839788 13172 log.cc:1196] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/be19b3d8910b41bd8c0c9b1aac96890e
I20260806 22:44:29.840440 13172 ts_tablet_manager.cc:1939] T be19b3d8910b41bd8c0c9b1aac96890e P 4537b6cc502a4ff3a548c7a7125c9734: Deleting consensus metadata
I20260806 22:44:29.840969 12704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:29.842795 12381 catalog_manager.cc:4841] TS 4537b6cc502a4ff3a548c7a7125c9734 (127.23.100.130:36179): tablet be19b3d8910b41bd8c0c9b1aac96890e (table test-table [id=eb781bfe10cd41fe8eaa02974d4efe5b]) successfully deleted
I20260806 22:44:29.844025 13173 ts_tablet_manager.cc:1918] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:29.844281 13173 log.cc:1196] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/be19b3d8910b41bd8c0c9b1aac96890e
I20260806 22:44:29.844988 13173 ts_tablet_manager.cc:1939] T be19b3d8910b41bd8c0c9b1aac96890e P 04cc6269cc1445528576a42ce5c1ef66: Deleting consensus metadata
I20260806 22:44:29.847288 12378 catalog_manager.cc:4841] TS 04cc6269cc1445528576a42ce5c1ef66 (127.23.100.131:42519): tablet be19b3d8910b41bd8c0c9b1aac96890e (table test-table [id=eb781bfe10cd41fe8eaa02974d4efe5b]) successfully deleted
I20260806 22:44:29.965355 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 12508
W20260806 22:44:29.991809 12378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.100.129:43837 (error 108)
I20260806 22:44:29.992163 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 12717
I20260806 22:44:30.003571 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:30.003069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43837 (local address 127.0.0.1:41082)
0806 22:44:30.003244 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:30.003381 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 22:44:30.004127 12378 catalog_manager.cc:4584] TS 54fdc651763245718a8b625e727046a5 (127.23.100.129:43837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be19b3d8910b41bd8c0c9b1aac96890e: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43837: connect: Connection refused (error 111)
Aug 06 22:44:30 dist-test-slave-zr1t krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:44:30.015724 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:29.993142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.23.100.130:36179)
0806 22:44:29.993689 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:29.993697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:29.993722 (+ 25us) server_negotiation.cc:407] Connection header received
0806 22:44:29.993785 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:29.993793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:29.993908 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:29.994019 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:29.996488 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:29.998390 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:30.001147 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:30.002108 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:30.002292 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:30.008103 (+ 5811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:30.008121 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:30.008139 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:30.008195 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:30.012434 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:30.013383 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:30.013392 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:30.013396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:30.013604 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:30.014066 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:30.014071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:30.014074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:30.014523 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:30.014832 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:30.015171 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 22:44:30.015348 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338}
I20260806 22:44:30.016517 12904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:30.058259 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:30.057760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43837 (local address 127.0.0.1:41086)
0806 22:44:30.057994 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:30.058092 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 22:44:30.059401 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:30.059057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43837 (local address 127.0.0.1:41092)
0806 22:44:30.059202 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:30.059274 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 22:44:30.120007 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:30.119505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43837 (local address 127.0.0.1:41098)
0806 22:44:30.119723 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:30.119818 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 22:44:30.121196 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:30.120775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43837 (local address 127.0.0.1:41102)
0806 22:44:30.120956 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:30.121052 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 22:44:30.134266 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 12717
I20260806 22:44:30.167026 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 12923
Aug 06 22:44:30 dist-test-slave-zr1t krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:44:30.183372 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:30.167794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.23.100.131:42519)
0806 22:44:30.168018 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:30.168024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:30.168046 (+ 22us) server_negotiation.cc:407] Connection header received
0806 22:44:30.168185 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:30.168189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:30.168258 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:30.168346 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:30.169784 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:30.170664 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:30.171924 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:30.172358 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:30.172469 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:30.177117 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:30.177134 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:30.177151 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:30.177196 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:30.180208 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:30.181050 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:30.181058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:30.181060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:30.181224 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:30.181836 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:30.181840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:30.181842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:30.182253 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:30.182531 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:30.182915 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 22:44:30.183107 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 22:44:30.184206 13099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:30.202004 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:30.201417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43837 (local address 127.0.0.1:41108)
0806 22:44:30.201727 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:30.201825 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260806 22:44:30.203199 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:30.202760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43837 (local address 127.0.0.1:41122)
0806 22:44:30.202995 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:30.203080 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260806 22:44:30.278090 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 12923
I20260806 22:44:30.305253 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 12349
I20260806 22:44:30.306910 12450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:30.422720 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:30.422166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43837 (local address 127.0.0.1:38560)
0806 22:44:30.422415 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:30.422535 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 22:44:30.422848 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 12349
I20260806 22:44:30.453943 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:36257
--webserver_interface=127.23.100.190
--webserver_port=41497
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:36257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:30.808418 13181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:30.808759 13181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:30.808976 13181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:30.817098 13181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:44:30.817209 13181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:30.817309 13181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:30.817422 13181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:44:30.817505 13181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:44:30.827824 13181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:36257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=41497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:30.829156 13181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:30.830576 13181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:30.860052 13189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:30.860338 13191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:30.860047 13198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:30.860808 13181 server_base.cc:1034] running on GCE node
I20260806 22:44:30.861932 13181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:30.863818 13181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:30.865149 13181 hybrid_clock.cc:648] HybridClock initialized: now 1786056270865094 us; error 35 us; skew 500 ppm
I20260806 22:44:30.865712 13181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:30 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056270, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:30.869622 13181 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:44:30.871626 13181 webserver.cc:460] Webserver started at http://127.23.100.190:41497/ using document root <none> and password file <none>
I20260806 22:44:30.872548 13181 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:30.872740 13181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:30.879276 13181 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 22:44:30.882368 13206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:30.883874 13181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:44:30.884111 13181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "1c09fa0a3e8043baa7b70a074235a6f8"
format_stamp: "Formatted at 2026-08-06 22:44:27 on dist-test-slave-zr1t"
server_key: "dcdebb2abf2841d4506148ae51eeed90"
server_key_iv: "1e9e891f8064d3621fdd983d560e667f"
server_key_version: "encryptionkey@0"
I20260806 22:44:30.885246 13181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:30.934607 13181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:30.939457 13181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:30.939786 13181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:30.960567 13181 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:36257
I20260806 22:44:30.960574 13281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:36257 every 8 connection(s)
I20260806 22:44:30.962312 13181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:44:30.966499 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 13181
I20260806 22:44:30.967393 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:43837
--local_ip_for_outbound_sockets=127.23.100.129
--tserver_master_addrs=127.23.100.190:36257
--webserver_port=34449
--webserver_interface=127.23.100.129
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:44:30.970921 13283 sys_catalog.cc:263] Verifying existing consensus state
I20260806 22:44:30.974843 13283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Bootstrap starting.
I20260806 22:44:31.005028 13283 log.cc:824] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:31.021508 13283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 22:44:31.022279 13283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Bootstrap complete.
I20260806 22:44:31.032852 13283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:31.033934 13283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c09fa0a3e8043baa7b70a074235a6f8, State: Initialized, Role: FOLLOWER
I20260806 22:44:31.034721 13283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:31.035006 13283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:44:31.035152 13283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:44:31.035351 13283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 22:44:31.039776 13283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:31.040349 13283 leader_election.cc:302] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c09fa0a3e8043baa7b70a074235a6f8; no voters:
I20260806 22:44:31.041110 13283 leader_election.cc:288] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 22:44:31.041272 13289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 22:44:31.043660 13289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 2 LEADER]: Becoming Leader. State: Replica: 1c09fa0a3e8043baa7b70a074235a6f8, State: Running, Role: LEADER
I20260806 22:44:31.043864 13283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:44:31.044523 13289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:31.050374 13292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } } }
I20260806 22:44:31.050798 13292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:31.051470 13293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c09fa0a3e8043baa7b70a074235a6f8. Latest consensus state: current_term: 2 leader_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } } }
I20260806 22:44:31.055102 13293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:31.062428 13297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:44:31.073755 13297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:44:31.075884 13297 catalog_manager.cc:1259] Loaded cluster ID: 0038493f127c4991abdae7edf798f59b
I20260806 22:44:31.076031 13297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:44:31.080996 13297 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 22:44:31.110422 13297 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 22:44:31.131788 13297 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:44:31.134521 13297 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Loaded TSK: 0
I20260806 22:44:31.136844 13297 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 22:44:31.376210 13287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:31.376507 13287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:31.376653 13287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:31.384256 13287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:31.384377 13287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:31.384546 13287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:44:31.395113 13287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:43837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=34449
--tserver_master_addrs=127.23.100.190:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:31.396464 13287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:31.397909 13287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:31.413806 13318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:31.413892 13319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:31.415464 13323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:31.416146 13287 server_base.cc:1034] running on GCE node
I20260806 22:44:31.416898 13287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:31.418072 13287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:31.419320 13287 hybrid_clock.cc:648] HybridClock initialized: now 1786056271419245 us; error 62 us; skew 500 ppm
I20260806 22:44:31.419674 13287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:31 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056271, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:31.423069 13287 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:44:31.424767 13287 webserver.cc:460] Webserver started at http://127.23.100.129:34449/ using document root <none> and password file <none>
I20260806 22:44:31.425503 13287 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:31.425709 13287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:31.431479 13287 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 22:44:31.434777 13334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:31.436437 13287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:44:31.436645 13287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "54fdc651763245718a8b625e727046a5"
format_stamp: "Formatted at 2026-08-06 22:44:28 on dist-test-slave-zr1t"
server_key: "e09aedc9b2f847154092bd502b780c18"
server_key_iv: "1824a47c3799250bd6156c8a49f94ef3"
server_key_version: "encryptionkey@0"
I20260806 22:44:31.437708 13287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:31.456151 13287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:31.460724 13287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:31.461023 13287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:31.462132 13287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:31.464257 13347 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 22:44:31.473160 13287 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 22:44:31.473387 13287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20260806 22:44:31.473600 13287 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 22:44:31.476914 13287 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 22:44:31.477038 13287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260806 22:44:31.477433 13347 tablet_bootstrap.cc:492] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Bootstrap starting.
I20260806 22:44:31.487146 13347 log.cc:824] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:31.506182 13347 tablet_bootstrap.cc:492] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 22:44:31.507781 13347 tablet_bootstrap.cc:492] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Bootstrap complete.
I20260806 22:44:31.508423 13347 ts_tablet_manager.cc:1397] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.005s
I20260806 22:44:31.520105 13347 raft_consensus.cc:348] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:31.521214 13347 raft_consensus.cc:729] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54fdc651763245718a8b625e727046a5, State: Initialized, Role: FOLLOWER
I20260806 22:44:31.522168 13347 consensus_queue.cc:260] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:31.524242 13347 ts_tablet_manager.cc:1428] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
I20260806 22:44:31.535419 13287 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:43837
I20260806 22:44:31.535511 13509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:43837 every 8 connection(s)
I20260806 22:44:31.536712 13287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:44:31.539389 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 13287
I20260806 22:44:31.540393 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:36179
--local_ip_for_outbound_sockets=127.23.100.130
--tserver_master_addrs=127.23.100.190:36257
--webserver_port=36927
--webserver_interface=127.23.100.130
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:44:31 dist-test-slave-zr1t krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056271, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:31.566540 13514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:31.539924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:56561)
0806 22:44:31.540827 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:31.540879 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:31.541402 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:31.547071 (+ 5669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:31.547083 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:31.547908 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:31.548555 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:31.548576 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:31.550491 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:31.550497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:31.551863 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:31.551877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:31.552944 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:31.552952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:31.553261 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:31.554084 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:31.554114 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:31.556834 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:31.562118 (+ 5284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:31.562126 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:31.562137 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:31.562641 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:31.563214 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:31.563217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:31.563219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:31.563399 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:31.564503 (+ 1104us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:31.564509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:31.564806 (+ 297us) client_negotiation.cc:769] Sending connection context
0806 22:44:31.565084 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 22:44:31.565473 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":397,"threads_started":1}
I20260806 22:44:31.567276 13516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:31.540023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:56561 (local address 127.23.100.190:36257)
0806 22:44:31.545446 (+ 5423us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:31.545461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:31.545509 (+ 48us) server_negotiation.cc:407] Connection header received
0806 22:44:31.546011 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:31.546026 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:31.546460 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:31.546888 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:31.548834 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:31.550299 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:31.552039 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:31.552715 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:31.552941 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:31.557197 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:31.557211 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:31.557235 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:31.557306 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:31.561951 (+ 4645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:31.562805 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:31.562819 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:31.562830 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:31.563077 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:31.563522 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:31.563530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:31.563537 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:31.564353 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:31.564556 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:31.565420 (+ 864us) server_negotiation.cc:299] Negotiation successful
0806 22:44:31.565934 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5182,"thread_start_us":5010,"threads_started":1}
I20260806 22:44:31.570169 13510 heartbeater.cc:347] Connected to a master server at 127.23.100.190:36257
I20260806 22:44:31.570505 13510 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:31.571236 13510 heartbeater.cc:511] Master 127.23.100.190:36257 requested a full tablet report, sending...
I20260806 22:44:31.574472 13232 ts_manager.cc:194] Registered new tserver with Master: 54fdc651763245718a8b625e727046a5 (127.23.100.129:43837)
I20260806 22:44:31.579118 13232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:56561
I20260806 22:44:31.593134 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:31.578206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.23.100.129:43837)
0806 22:44:31.584904 (+ 6698us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:31.584912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:31.585621 (+ 709us) server_negotiation.cc:407] Connection header received
0806 22:44:31.585731 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:31.585737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:31.585908 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:31.586064 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:31.586723 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:31.587821 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:31.590837 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:31.591903 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:31.592102 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:31.592252 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:31.592728 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 22:44:31.592856 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6483,"thread_start_us":6274,"threads_started":1}
I20260806 22:44:31.593398 13519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:31.578020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43837 (local address 127.0.0.1:38572)
0806 22:44:31.585248 (+ 7228us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:31.585299 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:31.585493 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:31.586117 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:31.586125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:31.586159 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:31.586541 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:31.586554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:31.587972 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:31.587979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:31.590666 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:31.590686 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:31.592167 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:31.592174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:31.592413 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:31.592486 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 22:44:31.592666 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 22:44:31.592896 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6985,"thread_start_us":6823,"threads_started":1}
I20260806 22:44:31.595223 13415 tablet_service.cc:1514] Processing DeleteTablet for tablet be19b3d8910b41bd8c0c9b1aac96890e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:29 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:38572
I20260806 22:44:31.597420 13522 tablet_replica.cc:330] stopping tablet replica
I20260806 22:44:31.597831 13522 raft_consensus.cc:2229] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:44:31.598222 13522 raft_consensus.cc:2258] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:31.599444 13522 ts_tablet_manager.cc:1905] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:31.606817 13522 ts_tablet_manager.cc:1918] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:31.606977 13522 log.cc:1196] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/be19b3d8910b41bd8c0c9b1aac96890e
I20260806 22:44:31.607431 13522 ts_tablet_manager.cc:1939] T be19b3d8910b41bd8c0c9b1aac96890e P 54fdc651763245718a8b625e727046a5: Deleting consensus metadata
I20260806 22:44:31.609472 13209 catalog_manager.cc:4841] TS 54fdc651763245718a8b625e727046a5 (127.23.100.129:43837): tablet be19b3d8910b41bd8c0c9b1aac96890e (table test-table [id=eb781bfe10cd41fe8eaa02974d4efe5b]) successfully deleted
W20260806 22:44:31.913085 13517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:31.913460 13517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:31.913646 13517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:31.921203 13517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:31.921373 13517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:31.921623 13517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:44:31.931684 13517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:36179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=36927
--tserver_master_addrs=127.23.100.190:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:31.933020 13517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:31.934507 13517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:31.948571 13528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:31.948549 13534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:31.948544 13530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:31.949785 13517 server_base.cc:1034] running on GCE node
I20260806 22:44:31.950523 13517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:31.951531 13517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:31.952775 13517 hybrid_clock.cc:648] HybridClock initialized: now 1786056271952707 us; error 59 us; skew 500 ppm
I20260806 22:44:31.953125 13517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:31 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056271, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:31.956727 13517 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:44:31.958474 13517 webserver.cc:460] Webserver started at http://127.23.100.130:36927/ using document root <none> and password file <none>
I20260806 22:44:31.959239 13517 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:31.959427 13517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:31.967141 13517 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:44:31.970911 13548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:31.972697 13517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 22:44:31.972880 13517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "4537b6cc502a4ff3a548c7a7125c9734"
format_stamp: "Formatted at 2026-08-06 22:44:28 on dist-test-slave-zr1t"
server_key: "5a1d2e0155f055c0c33821934c203985"
server_key_iv: "57180a39fcbc5a97a8d89d8b0f42cab7"
server_key_version: "encryptionkey@0"
I20260806 22:44:31.974373 13517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:32.018656 13517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:32.023705 13517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:32.023977 13517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:32.025099 13517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:32.026701 13517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:32.026818 13517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:32.026906 13517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:32.026968 13517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:32.064541 13517 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:36179
I20260806 22:44:32.064657 13717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:36179 every 8 connection(s)
I20260806 22:44:32.066291 13517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:44:32.070983 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 13517
I20260806 22:44:32.071640 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:42519
--local_ip_for_outbound_sockets=127.23.100.131
--tserver_master_addrs=127.23.100.190:36257
--webserver_port=35819
--webserver_interface=127.23.100.131
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:44:32 dist-test-slave-zr1t krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056271, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:32.089334 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.069241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:39853 (local address 127.23.100.190:36257)
0806 22:44:32.069735 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:32.069740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:32.070788 (+ 1048us) server_negotiation.cc:407] Connection header received
0806 22:44:32.073069 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:32.073076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:32.073192 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:32.073328 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:32.075073 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.075884 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.077778 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.078252 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.078425 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.082533 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:32.082538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:32.082542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:32.082582 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:32.085738 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:32.086650 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:32.086655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:32.086657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:32.086797 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:32.087308 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:32.087312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:32.087315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:32.087617 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:32.087795 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:32.088648 (+ 853us) server_negotiation.cc:299] Negotiation successful
0806 22:44:32.088902 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":216,"threads_started":1}
I20260806 22:44:32.090619 13723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.069839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:39853)
0806 22:44:32.070606 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:32.070648 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:32.072724 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:32.073665 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:32.073674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:32.074248 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:32.074858 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.074889 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.076075 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.076080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.077607 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.077622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.078517 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.078525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.078797 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.079628 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:32.079654 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:32.082269 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:32.085923 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:32.085933 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:32.085952 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:32.086488 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:32.086943 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:32.086949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:32.086953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:32.087161 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:32.087766 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:32.087776 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:32.088164 (+ 388us) client_negotiation.cc:769] Sending connection context
0806 22:44:32.088525 (+ 361us) client_negotiation.cc:240] Negotiation successful
0806 22:44:32.089165 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":280,"threads_started":1}
I20260806 22:44:32.092363 13719 heartbeater.cc:347] Connected to a master server at 127.23.100.190:36257
I20260806 22:44:32.092726 13719 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:32.093789 13719 heartbeater.cc:511] Master 127.23.100.190:36257 requested a full tablet report, sending...
I20260806 22:44:32.095479 13232 ts_manager.cc:194] Registered new tserver with Master: 4537b6cc502a4ff3a548c7a7125c9734 (127.23.100.130:36179)
I20260806 22:44:32.096330 13232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:39853
W20260806 22:44:32.420368 13724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:32.420647 13724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:32.420804 13724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:32.428107 13724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:32.428220 13724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:32.428429 13724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:44:32.438253 13724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:42519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=35819
--tserver_master_addrs=127.23.100.190:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:32.439530 13724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:32.440845 13724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:32.454353 13732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:32.454443 13736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:32.454469 13733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:32.455319 13724 server_base.cc:1034] running on GCE node
I20260806 22:44:32.456051 13724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:32.457027 13724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:32.458293 13724 hybrid_clock.cc:648] HybridClock initialized: now 1786056272458237 us; error 62 us; skew 500 ppm
I20260806 22:44:32.458626 13724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:32 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056272, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:32.462006 13724 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:44:32.463586 13724 webserver.cc:460] Webserver started at http://127.23.100.131:35819/ using document root <none> and password file <none>
I20260806 22:44:32.464252 13724 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:32.464372 13724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:32.469822 13724 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 22:44:32.473019 13744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:32.474555 13724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:44:32.474761 13724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "04cc6269cc1445528576a42ce5c1ef66"
format_stamp: "Formatted at 2026-08-06 22:44:29 on dist-test-slave-zr1t"
server_key: "39a0fdcae37c0e6dd5ef14b554148cb0"
server_key_iv: "26f6020af0e5a42acccf20e189a85833"
server_key_version: "encryptionkey@0"
I20260806 22:44:32.475826 13724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:32.502413 13724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:32.507386 13724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:32.507691 13724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:32.508780 13724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:32.510309 13724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:32.510450 13724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:32.510581 13724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:32.510668 13724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:32.549635 13724 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:42519
I20260806 22:44:32.549697 13903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:42519 every 8 connection(s)
I20260806 22:44:32.551195 13724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:44:32.555718 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 13724
Aug 06 22:44:32 dist-test-slave-zr1t krb5kdc[12334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056272, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:32.572072 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.556736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.23.100.190:36257)
0806 22:44:32.557242 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:32.557249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:32.557689 (+ 440us) server_negotiation.cc:407] Connection header received
0806 22:44:32.557925 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:32.557930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:32.558005 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:32.558130 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:32.559812 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.560581 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.562124 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.562575 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.562717 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.565724 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:32.565729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:32.565733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:32.565766 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:32.569124 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:32.569995 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:32.570000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:32.570002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:32.570124 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:32.570551 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:32.570555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:32.570557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:32.570845 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:32.571129 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:32.571442 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 22:44:32.571687 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":370,"thread_start_us":276,"threads_started":1}
I20260806 22:44:32.572893 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.553888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:43611 (local address 127.23.100.190:36257)
0806 22:44:32.554124 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:32.554131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:32.555433 (+ 1302us) server_negotiation.cc:407] Connection header received
0806 22:44:32.558085 (+ 2652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:32.558091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:32.558173 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:32.558315 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:32.559962 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.560816 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.562542 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.562980 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.563120 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.566631 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:32.566637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:32.566641 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:32.566694 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:32.569708 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:32.570454 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:32.570459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:32.570464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:32.570615 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:32.571078 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:32.571082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:32.571084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:32.571348 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:32.571523 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:32.572326 (+ 803us) server_negotiation.cc:299] Negotiation successful
0806 22:44:32.572541 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":82,"spinlock_wait_cycles":12160}
I20260806 22:44:32.574052 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.554434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:43611)
0806 22:44:32.555236 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:32.555297 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:32.557790 (+ 2493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:32.558566 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:32.558575 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:32.559119 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:32.559773 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.559791 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.560997 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.561001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.562376 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.562391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.563218 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.563227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.563456 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.564075 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:32.564098 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:32.566402 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:32.569855 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:32.569864 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:32.569881 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:32.570321 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:32.570732 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:32.570738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:32.570741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:32.570945 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:32.571492 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:32.571501 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:32.571846 (+ 345us) client_negotiation.cc:769] Sending connection context
0806 22:44:32.572213 (+ 367us) client_negotiation.cc:240] Negotiation successful
0806 22:44:32.572774 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":321,"threads_started":1}
I20260806 22:44:32.575461 13905 heartbeater.cc:347] Connected to a master server at 127.23.100.190:36257
I20260806 22:44:32.575802 13905 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:32.576577 13905 heartbeater.cc:511] Master 127.23.100.190:36257 requested a full tablet report, sending...
I20260806 22:44:32.578104 13232 ts_manager.cc:194] Registered new tserver with Master: 04cc6269cc1445528576a42ce5c1ef66 (127.23.100.131:42519)
I20260806 22:44:32.578940 13232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:43611
I20260806 22:44:32.584535 13510 heartbeater.cc:503] Master 127.23.100.190:36257 was elected leader, sending a full tablet report...
I20260806 22:44:32.587688 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:44:32.595774 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.588794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.23.100.190:36257)
0806 22:44:32.589008 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:32.589017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:32.589634 (+ 617us) server_negotiation.cc:407] Connection header received
0806 22:44:32.589979 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:32.589988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:32.590201 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:32.590366 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:44:32.590822 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.591684 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.593304 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.593834 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.593982 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.594577 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:44:32.595113 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:44:32.595286 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:32.595372 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 22:44:32.595471 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 22:44:32.596875 13232 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:44:32.605218 13232 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:44:32.620134 13415 tablet_service.cc:1467] Processing CreateTablet for tablet 50d8057da9134357bbe45d9dd321da13 (DEFAULT_TABLE table=test-table [id=de58bcacd53245c9b6dfe167b2a8c304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:32.621299 13415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d8057da9134357bbe45d9dd321da13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:32.629634 13927 tablet_bootstrap.cc:492] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5: Bootstrap starting.
I20260806 22:44:32.631649 13927 tablet_bootstrap.cc:654] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:32.633072 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.619192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.23.100.131:42519)
0806 22:44:32.625088 (+ 5896us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:32.625097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:32.625131 (+ 34us) server_negotiation.cc:407] Connection header received
0806 22:44:32.625222 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:32.625229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:32.625437 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:32.625652 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:32.626315 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.628028 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.630534 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.631945 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.632180 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.632368 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:32.632561 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 22:44:32.632734 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5700,"thread_start_us":5536,"threads_started":1}
I20260806 22:44:32.633174 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.617691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:42519 (local address 127.0.0.1:48556)
0806 22:44:32.618765 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:32.618785 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:32.618909 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:32.625669 (+ 6760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:32.625674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:32.625699 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:32.626084 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.626100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.628172 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.628178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.630373 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.630384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.632155 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.632162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.632339 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.632374 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 22:44:32.632513 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 22:44:32.632677 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":1,"thread_start_us":541,"threads_started":1}
I20260806 22:44:32.634245 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.618730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.23.100.130:36179)
0806 22:44:32.624670 (+ 5940us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:32.624683 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:32.624727 (+ 44us) server_negotiation.cc:407] Connection header received
0806 22:44:32.624853 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:32.624862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:32.625130 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:32.625357 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:32.626032 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.627574 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.630362 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.631939 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.632144 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.632372 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:32.633508 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0806 22:44:32.633777 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5685,"thread_start_us":5513,"threads_started":1}
I20260806 22:44:32.634939 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.617395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:36179 (local address 127.0.0.1:46994)
0806 22:44:32.618366 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:32.618393 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:32.618539 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:32.625348 (+ 6809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:32.625353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:32.625383 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:32.625799 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.625819 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.627762 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.627766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.630154 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.630171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.632547 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.632554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.632742 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.632788 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 22:44:32.632891 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 22:44:32.633104 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":118,"thread_start_us":456,"threads_started":1}
I20260806 22:44:32.637545 13816 tablet_service.cc:1467] Processing CreateTablet for tablet 50d8057da9134357bbe45d9dd321da13 (DEFAULT_TABLE table=test-table [id=de58bcacd53245c9b6dfe167b2a8c304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:32.638573 13621 tablet_service.cc:1467] Processing CreateTablet for tablet 50d8057da9134357bbe45d9dd321da13 (DEFAULT_TABLE table=test-table [id=de58bcacd53245c9b6dfe167b2a8c304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:32.639139 13816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d8057da9134357bbe45d9dd321da13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:32.639910 13621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d8057da9134357bbe45d9dd321da13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:32.640196 13927 tablet_bootstrap.cc:492] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5: No bootstrap required, opened a new log
I20260806 22:44:32.640381 13927 ts_tablet_manager.cc:1397] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260806 22:44:32.641099 13927 raft_consensus.cc:348] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:32.641333 13927 raft_consensus.cc:374] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:32.641498 13927 raft_consensus.cc:729] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54fdc651763245718a8b625e727046a5, State: Initialized, Role: FOLLOWER
I20260806 22:44:32.641819 13927 consensus_queue.cc:260] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:32.644315 13927 ts_tablet_manager.cc:1428] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260806 22:44:32.656466 13931 tablet_bootstrap.cc:492] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734: Bootstrap starting.
I20260806 22:44:32.657786 13933 tablet_bootstrap.cc:492] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66: Bootstrap starting.
I20260806 22:44:32.660848 13931 tablet_bootstrap.cc:654] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:32.661774 13933 tablet_bootstrap.cc:654] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:32.662416 13931 log.cc:824] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:32.663213 13933 log.cc:824] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:32.665622 13931 tablet_bootstrap.cc:492] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734: No bootstrap required, opened a new log
I20260806 22:44:32.665939 13931 ts_tablet_manager.cc:1397] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 22:44:32.666126 13933 tablet_bootstrap.cc:492] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66: No bootstrap required, opened a new log
I20260806 22:44:32.666385 13933 ts_tablet_manager.cc:1397] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260806 22:44:32.674520 13931 raft_consensus.cc:348] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:32.674520 13933 raft_consensus.cc:348] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:32.674953 13931 raft_consensus.cc:374] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:32.674961 13933 raft_consensus.cc:374] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:32.675072 13931 raft_consensus.cc:729] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4537b6cc502a4ff3a548c7a7125c9734, State: Initialized, Role: FOLLOWER
I20260806 22:44:32.675087 13933 raft_consensus.cc:729] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04cc6269cc1445528576a42ce5c1ef66, State: Initialized, Role: FOLLOWER
I20260806 22:44:32.675710 13933 consensus_queue.cc:260] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:32.675734 13931 consensus_queue.cc:260] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:32.677103 13719 heartbeater.cc:503] Master 127.23.100.190:36257 was elected leader, sending a full tablet report...
I20260806 22:44:32.677655 13905 heartbeater.cc:503] Master 127.23.100.190:36257 was elected leader, sending a full tablet report...
I20260806 22:44:32.677767 13931 ts_tablet_manager.cc:1428] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260806 22:44:32.678406 13933 ts_tablet_manager.cc:1428] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260806 22:44:32.698304 13930 raft_consensus.cc:482] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:44:32.698652 13930 raft_consensus.cc:504] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:32.700677 13930 leader_election.cc:288] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4537b6cc502a4ff3a548c7a7125c9734 (127.23.100.130:36179), 04cc6269cc1445528576a42ce5c1ef66 (127.23.100.131:42519)
I20260806 22:44:32.707623 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.701316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:57603 (local address 127.23.100.131:42519)
0806 22:44:32.701611 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:32.701620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:32.702235 (+ 615us) server_negotiation.cc:407] Connection header received
0806 22:44:32.702460 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:32.702464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:32.702522 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:32.702620 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:32.703217 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.703956 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.705893 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.706691 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.706883 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.706964 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:32.707227 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 22:44:32.707357 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 22:44:32.708000 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.701053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:42519 (local address 127.23.100.129:57603)
0806 22:44:32.702126 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:32.702140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:32.702305 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:32.702640 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:32.702645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:32.702673 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:32.703053 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.703062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.704099 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.704104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.705713 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.705727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.706875 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.706881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.707060 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.707084 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 22:44:32.707196 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 22:44:32.707335 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":6,"thread_start_us":774,"threads_started":1}
I20260806 22:44:32.708214 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.701174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:37447 (local address 127.23.100.130:36179)
0806 22:44:32.701416 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:32.701425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:32.701861 (+ 436us) server_negotiation.cc:407] Connection header received
0806 22:44:32.702111 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:32.702116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:32.702179 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:32.702358 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:32.703388 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.704182 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.706294 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.707123 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.707323 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.707407 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:32.707737 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 22:44:32.707922 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 22:44:32.708398 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.700981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:36179 (local address 127.23.100.129:37447)
0806 22:44:32.701728 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:32.701745 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:32.701943 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:32.702718 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:32.702722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:32.702750 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:32.703196 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.703205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.704339 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.704348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.706091 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:32.706105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.707363 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.707370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:32.707555 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.707580 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 22:44:32.707707 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 22:44:32.707879 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":67,"thread_start_us":448,"threads_started":1}
I20260806 22:44:32.709139 13845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d8057da9134357bbe45d9dd321da13" candidate_uuid: "54fdc651763245718a8b625e727046a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04cc6269cc1445528576a42ce5c1ef66" is_pre_election: true
I20260806 22:44:32.709676 13845 raft_consensus.cc:2454] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54fdc651763245718a8b625e727046a5 in term 0.
I20260806 22:44:32.709775 13652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d8057da9134357bbe45d9dd321da13" candidate_uuid: "54fdc651763245718a8b625e727046a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4537b6cc502a4ff3a548c7a7125c9734" is_pre_election: true
I20260806 22:44:32.710361 13652 raft_consensus.cc:2454] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54fdc651763245718a8b625e727046a5 in term 0.
I20260806 22:44:32.710400 13339 leader_election.cc:302] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04cc6269cc1445528576a42ce5c1ef66, 54fdc651763245718a8b625e727046a5; no voters:
I20260806 22:44:32.710860 13930 raft_consensus.cc:2781] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:44:32.711062 13930 raft_consensus.cc:482] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:44:32.711186 13930 raft_consensus.cc:3037] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:32.713728 13930 raft_consensus.cc:504] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:32.714391 13930 leader_election.cc:288] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [CANDIDATE]: Term 1 election: Requested vote from peers 4537b6cc502a4ff3a548c7a7125c9734 (127.23.100.130:36179), 04cc6269cc1445528576a42ce5c1ef66 (127.23.100.131:42519)
I20260806 22:44:32.714888 13652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d8057da9134357bbe45d9dd321da13" candidate_uuid: "54fdc651763245718a8b625e727046a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4537b6cc502a4ff3a548c7a7125c9734"
I20260806 22:44:32.715009 13845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d8057da9134357bbe45d9dd321da13" candidate_uuid: "54fdc651763245718a8b625e727046a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04cc6269cc1445528576a42ce5c1ef66"
I20260806 22:44:32.715137 13652 raft_consensus.cc:3037] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:32.715252 13845 raft_consensus.cc:3037] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:32.718947 13845 raft_consensus.cc:2454] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54fdc651763245718a8b625e727046a5 in term 1.
I20260806 22:44:32.718948 13652 raft_consensus.cc:2454] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54fdc651763245718a8b625e727046a5 in term 1.
I20260806 22:44:32.719460 13339 leader_election.cc:302] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04cc6269cc1445528576a42ce5c1ef66, 54fdc651763245718a8b625e727046a5; no voters:
I20260806 22:44:32.719882 13930 raft_consensus.cc:2781] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:44:32.720404 13930 raft_consensus.cc:686] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 1 LEADER]: Becoming Leader. State: Replica: 54fdc651763245718a8b625e727046a5, State: Running, Role: LEADER
I20260806 22:44:32.720804 13930 consensus_queue.cc:237] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } } peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } }
I20260806 22:44:32.725664 13232 catalog_manager.cc:5495] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54fdc651763245718a8b625e727046a5 (127.23.100.129). New cstate: current_term: 1 leader_uuid: "54fdc651763245718a8b625e727046a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54fdc651763245718a8b625e727046a5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43837 } health_report { overall_health: HEALTHY } } }
I20260806 22:44:32.759092 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.752329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.23.100.129:43837)
0806 22:44:32.753012 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:32.753018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:32.753038 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:44:32.753128 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:32.753132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:32.753216 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:32.753399 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:44:32.753926 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.754802 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.756472 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.756962 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.757118 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.757677 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:44:32.758295 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:44:32.758462 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:32.758601 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 22:44:32.758694 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":360,"threads_started":1}
I20260806 22:44:32.765359 13652 raft_consensus.cc:1261] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734 [term 1 FOLLOWER]: Refusing update from remote peer 54fdc651763245718a8b625e727046a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:44:32.765369 13845 raft_consensus.cc:1261] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66 [term 1 FOLLOWER]: Refusing update from remote peer 54fdc651763245718a8b625e727046a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:44:32.766395 13942 consensus_queue.cc:1035] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04cc6269cc1445528576a42ce5c1ef66" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:44:32.766749 13930 consensus_queue.cc:1035] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4537b6cc502a4ff3a548c7a7125c9734" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:44:32.778398 13950 mvcc.cc:205] Tried to move back new op lower bound from 7315686493237653504 to 7315686493068402688. Current Snapshot: MvccSnapshot[applied={T|T < 7315686493237653504}]
I20260806 22:44:32.784924 13951 mvcc.cc:205] Tried to move back new op lower bound from 7315686493237653504 to 7315686493068402688. Current Snapshot: MvccSnapshot[applied={T|T < 7315686493237653504}]
I20260806 22:44:32.796954 13232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:44:32.797237 13232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:44:32.800976 13232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Sending DeleteTablet for 3 replicas of tablet 50d8057da9134357bbe45d9dd321da13
I20260806 22:44:32.802019 13415 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d8057da9134357bbe45d9dd321da13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:32 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:38572
I20260806 22:44:32.802070 13621 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d8057da9134357bbe45d9dd321da13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:32 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:46994
I20260806 22:44:32.802158 13816 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d8057da9134357bbe45d9dd321da13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:32 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:48556
I20260806 22:44:32.803108 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 13287
I20260806 22:44:32.803960 13967 tablet_replica.cc:330] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5: stopping tablet replica
I20260806 22:44:32.804234 13967 raft_consensus.cc:2229] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:44:32.804579 13969 tablet_replica.cc:330] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734: stopping tablet replica
I20260806 22:44:32.804915 13967 raft_consensus.cc:2258] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:32.805022 13969 raft_consensus.cc:2229] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:44:32.805590 13969 raft_consensus.cc:2258] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:32.806406 13971 tablet_replica.cc:330] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66: stopping tablet replica
I20260806 22:44:32.806825 13971 raft_consensus.cc:2229] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:44:32.806910 13967 ts_tablet_manager.cc:1905] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:32.807338 13971 raft_consensus.cc:2258] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:32.810063 13971 ts_tablet_manager.cc:1905] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:32.811090 13969 ts_tablet_manager.cc:1905] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:32.814460 13967 ts_tablet_manager.cc:1918] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:32.814679 13967 log.cc:1196] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/50d8057da9134357bbe45d9dd321da13
I20260806 22:44:32.815269 13967 ts_tablet_manager.cc:1939] T 50d8057da9134357bbe45d9dd321da13 P 54fdc651763245718a8b625e727046a5: Deleting consensus metadata
I20260806 22:44:32.816563 13209 catalog_manager.cc:4841] TS 54fdc651763245718a8b625e727046a5 (127.23.100.129:43837): tablet 50d8057da9134357bbe45d9dd321da13 (table test-table [id=de58bcacd53245c9b6dfe167b2a8c304]) successfully deleted
I20260806 22:44:32.817406 13971 ts_tablet_manager.cc:1918] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:32.817633 13971 log.cc:1196] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/50d8057da9134357bbe45d9dd321da13
I20260806 22:44:32.818261 13971 ts_tablet_manager.cc:1939] T 50d8057da9134357bbe45d9dd321da13 P 04cc6269cc1445528576a42ce5c1ef66: Deleting consensus metadata
I20260806 22:44:32.820137 13209 catalog_manager.cc:4841] TS 04cc6269cc1445528576a42ce5c1ef66 (127.23.100.131:42519): tablet 50d8057da9134357bbe45d9dd321da13 (table test-table [id=de58bcacd53245c9b6dfe167b2a8c304]) successfully deleted
I20260806 22:44:32.821233 13969 ts_tablet_manager.cc:1918] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:32.821420 13969 log.cc:1196] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/50d8057da9134357bbe45d9dd321da13
I20260806 22:44:32.821915 13969 ts_tablet_manager.cc:1939] T 50d8057da9134357bbe45d9dd321da13 P 4537b6cc502a4ff3a548c7a7125c9734: Deleting consensus metadata
I20260806 22:44:32.823539 13212 catalog_manager.cc:4841] TS 4537b6cc502a4ff3a548c7a7125c9734 (127.23.100.130:36179): tablet 50d8057da9134357bbe45d9dd321da13 (table test-table [id=de58bcacd53245c9b6dfe167b2a8c304]) successfully deleted
I20260806 22:44:32.831919 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.804679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.23.100.129:43837)
0806 22:44:32.804922 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:32.804931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:32.804956 (+ 25us) server_negotiation.cc:407] Connection header received
0806 22:44:32.805036 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:32.805044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:32.805142 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:32.805286 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:32.807973 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.809190 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.813286 (+ 4096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.814063 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.814204 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.825094 (+ 10890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:32.825110 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:32.825124 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:32.825165 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:32.828410 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:32.829298 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:32.829305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:32.829308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:32.829442 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:32.829998 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:32.830003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:32.830005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:32.830443 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:32.830721 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:32.831157 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 22:44:32.831447 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 22:44:32.832726 13498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:32.933841 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 13287
I20260806 22:44:32.961987 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 13517
I20260806 22:44:32.978164 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:32.962988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.23.100.130:36179)
0806 22:44:32.963218 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:32.963224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:32.963248 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:44:32.963363 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:32.963369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:32.963455 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:32.963570 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:32.964937 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.966101 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.967456 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:32.968121 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:32.968245 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:32.971245 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:32.971275 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:32.971297 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:32.971346 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:32.974858 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:32.975830 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:32.975838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:32.975841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:32.976002 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:32.976435 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:32.976439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:32.976441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:32.976834 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:32.977081 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:32.977450 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 22:44:32.977736 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 22:44:32.978859 13712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:33.072983 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 13517
I20260806 22:44:33.099826 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 13724
I20260806 22:44:33.114580 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:33.100724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.23.100.131:42519)
0806 22:44:33.100973 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:33.100980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:33.101004 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:44:33.101205 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:33.101210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:33.101276 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:33.101387 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:33.102883 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:33.103641 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:33.105051 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:33.105495 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:33.105724 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:33.108509 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:33.108526 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:33.108542 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:33.108594 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:33.111366 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:33.112145 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:33.112153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:33.112156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:33.112301 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:33.112705 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:33.112711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:33.112716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:33.113095 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:33.113413 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:33.113962 (+ 549us) server_negotiation.cc:299] Negotiation successful
0806 22:44:33.114225 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 22:44:33.115326 13897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:33.207162 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 13724
I20260806 22:44:33.231163 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 13181
I20260806 22:44:33.232446 13274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:33.349731 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 13181
I20260806 22:44:33.380075 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:36257
--webserver_interface=127.23.100.190
--webserver_port=41497
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:36257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:33.731997 13979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:33.732311 13979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:33.732514 13979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:33.740175 13979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:44:33.740301 13979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:33.740401 13979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:33.740499 13979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:44:33.740562 13979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:44:33.751124 13979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:36257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=41497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:33.752535 13979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:33.754001 13979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:33.766607 13984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:33.766659 13986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:33.766563 13988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:33.768014 13979 server_base.cc:1034] running on GCE node
I20260806 22:44:33.768846 13979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:33.770421 13979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:33.771683 13979 hybrid_clock.cc:648] HybridClock initialized: now 1786056273771643 us; error 56 us; skew 500 ppm
I20260806 22:44:33.772053 13979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:33 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056273, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:33.775504 13979 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:44:33.777099 13979 webserver.cc:460] Webserver started at http://127.23.100.190:41497/ using document root <none> and password file <none>
I20260806 22:44:33.778146 13979 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:33.778270 13979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:33.784176 13979 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 22:44:33.787434 13998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:33.788870 13979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 22:44:33.789029 13979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "1c09fa0a3e8043baa7b70a074235a6f8"
format_stamp: "Formatted at 2026-08-06 22:44:27 on dist-test-slave-zr1t"
server_key: "dcdebb2abf2841d4506148ae51eeed90"
server_key_iv: "1e9e891f8064d3621fdd983d560e667f"
server_key_version: "encryptionkey@0"
I20260806 22:44:33.790110 13979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:33.818657 13979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:33.823385 13979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:33.823627 13979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:33.844161 13979 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:36257
I20260806 22:44:33.844180 14072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:36257 every 8 connection(s)
I20260806 22:44:33.845772 13979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:44:33.852200 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 13979
I20260806 22:44:33.852696 14073 sys_catalog.cc:263] Verifying existing consensus state
I20260806 22:44:33.853175 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:43837
--local_ip_for_outbound_sockets=127.23.100.129
--tserver_master_addrs=127.23.100.190:36257
--webserver_port=34449
--webserver_interface=127.23.100.129
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:44:33.856067 14073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Bootstrap starting.
I20260806 22:44:33.876396 14073 log.cc:824] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:33.892822 14073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 22:44:33.893537 14073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8: Bootstrap complete.
I20260806 22:44:33.903224 14073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:33.904182 14073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c09fa0a3e8043baa7b70a074235a6f8, State: Initialized, Role: FOLLOWER
I20260806 22:44:33.904870 14073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:33.905138 14073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:44:33.905263 14073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:44:33.905467 14073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 22:44:33.909704 14073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:33.910281 14073 leader_election.cc:302] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c09fa0a3e8043baa7b70a074235a6f8; no voters:
I20260806 22:44:33.911010 14073 leader_election.cc:288] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 22:44:33.911242 14079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 22:44:33.912369 14079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [term 3 LEADER]: Becoming Leader. State: Replica: 1c09fa0a3e8043baa7b70a074235a6f8, State: Running, Role: LEADER
I20260806 22:44:33.913215 14073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:44:33.913190 14079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } }
I20260806 22:44:33.923866 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c09fa0a3e8043baa7b70a074235a6f8. Latest consensus state: current_term: 3 leader_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } } }
I20260806 22:44:33.924357 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:33.926801 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09fa0a3e8043baa7b70a074235a6f8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 36257 } } }
I20260806 22:44:33.927129 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c09fa0a3e8043baa7b70a074235a6f8 [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:33.930368 14096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:44:33.939111 14096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:44:33.940824 14096 catalog_manager.cc:1259] Loaded cluster ID: 0038493f127c4991abdae7edf798f59b
I20260806 22:44:33.941013 14096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 22:44:33.943320 14096 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd21417b9ad google::LogMessage::Fail() at ??:0
@ 0x7fd21417e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd21417b4de google::LogMessage::Flush() at ??:0
@ 0x7fd21417ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd229bae512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
I20260806 22:44:33.963474 14104 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fd229bab976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd229c33610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd229c33439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd216dc7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd216dca91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd216dca799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd216da5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd21a07f6db start_thread at ??:0
@ 0x7fd21301561f clone at ??:0
W20260806 22:44:34.282953 14077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:34.283239 14077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:34.283339 14077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:34.290926 14077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:34.291103 14077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:34.291244 14077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:44:34.301254 14077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:43837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=34449
--tserver_master_addrs=127.23.100.190:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:34.302690 14077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:34.304023 14077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:34.317763 14133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:34.317921 14137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:34.317965 14134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:34.319279 14077 server_base.cc:1034] running on GCE node
I20260806 22:44:34.319938 14077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:34.320921 14077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:34.322168 14077 hybrid_clock.cc:648] HybridClock initialized: now 1786056274322126 us; error 59 us; skew 500 ppm
I20260806 22:44:34.322523 14077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:34 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056274, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:34.326006 14077 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:44:34.327540 14077 webserver.cc:460] Webserver started at http://127.23.100.129:34449/ using document root <none> and password file <none>
I20260806 22:44:34.328256 14077 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:34.328402 14077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:34.333954 14077 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 22:44:34.336815 14145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:34.338196 14077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:44:34.338397 14077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "54fdc651763245718a8b625e727046a5"
format_stamp: "Formatted at 2026-08-06 22:44:28 on dist-test-slave-zr1t"
server_key: "e09aedc9b2f847154092bd502b780c18"
server_key_iv: "1824a47c3799250bd6156c8a49f94ef3"
server_key_version: "encryptionkey@0"
I20260806 22:44:34.339397 14077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:34.362684 14077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:34.366972 14077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:34.367300 14077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:34.368374 14077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:34.369822 14077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:34.369975 14077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:34.370103 14077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:34.370245 14077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:34.407146 14077 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:43837
I20260806 22:44:34.407233 14298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:43837 every 8 connection(s)
I20260806 22:44:34.408689 14077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:44:34.414227 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:34.412107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:49763)
0806 22:44:34.412960 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:34.413203 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":391,"threads_started":1}
W20260806 22:44:34.414978 14299 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:36257 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.190:36257: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
I20260806 22:44:34.415969 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:34.415550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:41149)
0806 22:44:34.415702 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:34.415807 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 22:44:34.416514 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 14077
I20260806 22:44:34.417239 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:34.416787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:47777)
0806 22:44:34.416964 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:34.417070 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 22:44:34.417371 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:36179
--local_ip_for_outbound_sockets=127.23.100.130
--tserver_master_addrs=127.23.100.190:36257
--webserver_port=36927
--webserver_interface=127.23.100.130
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:34.417611 14299 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 22:44:34.765105 14306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:34.765467 14306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:34.765662 14306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:34.773074 14306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:34.773247 14306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:34.773440 14306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:44:34.783419 14306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:36179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=36927
--tserver_master_addrs=127.23.100.190:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:34.784777 14306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:34.786177 14306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:34.799551 14313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:34.799685 14315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:34.799746 14321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:34.800858 14306 server_base.cc:1034] running on GCE node
I20260806 22:44:34.801591 14306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:34.802577 14306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:34.803843 14306 hybrid_clock.cc:648] HybridClock initialized: now 1786056274803793 us; error 54 us; skew 500 ppm
I20260806 22:44:34.804205 14306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:34 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056274, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:34.807649 14306 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:44:34.809242 14306 webserver.cc:460] Webserver started at http://127.23.100.130:36927/ using document root <none> and password file <none>
I20260806 22:44:34.810026 14306 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:34.810185 14306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:34.815706 14306 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 22:44:34.818648 14336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:34.819972 14306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:44:34.820163 14306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "4537b6cc502a4ff3a548c7a7125c9734"
format_stamp: "Formatted at 2026-08-06 22:44:28 on dist-test-slave-zr1t"
server_key: "5a1d2e0155f055c0c33821934c203985"
server_key_iv: "57180a39fcbc5a97a8d89d8b0f42cab7"
server_key_version: "encryptionkey@0"
I20260806 22:44:34.821177 14306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:34.840333 14306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:34.845283 14306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:34.845634 14306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:34.846707 14306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:34.848095 14306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:34.848227 14306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:34.848373 14306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:34.848456 14306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:34.882503 14306 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:36179
I20260806 22:44:34.882601 14498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:36179 every 8 connection(s)
I20260806 22:44:34.884030 14306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:44:34.888207 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 14306
I20260806 22:44:34.889055 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:42519
--local_ip_for_outbound_sockets=127.23.100.131
--tserver_master_addrs=127.23.100.190:36257
--webserver_port=35819
--webserver_interface=127.23.100.131
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:44:34.889423 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:34.887037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:43117)
0806 22:44:34.887892 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:34.888192 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":309,"threads_started":1}
W20260806 22:44:34.890179 14500 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:36257 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.190:36257: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
I20260806 22:44:34.891294 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:34.890829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:57551)
0806 22:44:34.891036 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:34.891136 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 22:44:34.892356 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:34.891932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:51923)
0806 22:44:34.892101 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:34.892200 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 22:44:34.892612 14500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 22:44:35.232501 14505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:35.232786 14505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:35.232959 14505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:35.240295 14505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:35.240415 14505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:35.240612 14505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:44:35.250741 14505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:42519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=35819
--tserver_master_addrs=127.23.100.190:36257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:35.252071 14505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:35.253422 14505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:35.266561 14512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:35.266660 14514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:35.266966 14516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:35.267871 14505 server_base.cc:1034] running on GCE node
I20260806 22:44:35.268548 14505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:35.269596 14505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:35.270840 14505 hybrid_clock.cc:648] HybridClock initialized: now 1786056275270786 us; error 47 us; skew 500 ppm
I20260806 22:44:35.271183 14505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:35 dist-test-slave-zr1t krb5kdc[12334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056275, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:35.274598 14505 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:44:35.276083 14505 webserver.cc:460] Webserver started at http://127.23.100.131:35819/ using document root <none> and password file <none>
I20260806 22:44:35.276782 14505 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:35.276937 14505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:35.282469 14505 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:44:35.285121 14525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:35.286414 14505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:44:35.286602 14505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "04cc6269cc1445528576a42ce5c1ef66"
format_stamp: "Formatted at 2026-08-06 22:44:29 on dist-test-slave-zr1t"
server_key: "39a0fdcae37c0e6dd5ef14b554148cb0"
server_key_iv: "26f6020af0e5a42acccf20e189a85833"
server_key_version: "encryptionkey@0"
I20260806 22:44:35.287585 14505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:35.303264 14505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:35.307413 14505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:35.307720 14505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:35.308759 14505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:35.310328 14505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:35.310458 14505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:35.310599 14505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:35.310693 14505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:35.345283 14505 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:42519
I20260806 22:44:35.345379 14688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:42519 every 8 connection(s)
I20260806 22:44:35.346833 14505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:44:35.350128 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 14505
I20260806 22:44:35.352270 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:35.349962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:42539)
0806 22:44:35.350808 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:35.351114 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":323,"threads_started":1}
W20260806 22:44:35.353057 14691 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:36257 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.190:36257: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
I20260806 22:44:35.353987 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:35.353604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:43303)
0806 22:44:35.353785 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:35.353862 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 22:44:35.354854 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:35.354539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:52213)
0806 22:44:35.354662 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:35.354746 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 22:44:35.355123 14691 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 22:44:35.419186 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:35.418159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:53165)
0806 22:44:35.418677 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:35.418846 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
I20260806 22:44:35.894182 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:35.893207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:55359)
0806 22:44:35.893807 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:35.893948 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":318,"threads_started":1}
I20260806 22:44:36.356554 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:36.355662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:45037)
0806 22:44:36.356154 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:36.356294 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":245,"threads_started":1}
I20260806 22:44:36.420876 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:36.420169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:52131)
0806 22:44:36.420629 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:36.420714 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":243,"threads_started":1}
I20260806 22:44:36.896238 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:36.895221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:55565)
0806 22:44:36.895899 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:36.895989 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":360,"threads_started":1}
I20260806 22:44:37.358549 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:37.357665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:58409)
0806 22:44:37.358249 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:37.358375 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":321,"threads_started":1}
I20260806 22:44:37.422729 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:37.421927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:35665)
0806 22:44:37.422436 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:37.422530 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":263,"threads_started":1}
I20260806 22:44:37.898344 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:37.897406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:42849)
0806 22:44:37.898049 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:37.898154 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":404,"threads_started":1}
I20260806 22:44:38.360497 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:38.359636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:58197)
0806 22:44:38.360178 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:38.360303 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":281,"threads_started":1}
I20260806 22:44:38.424388 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:38.423658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:54657)
0806 22:44:38.424112 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:38.424196 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260806 22:44:38.900066 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:38.899280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:46093)
0806 22:44:38.899774 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:38.899862 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":262,"threads_started":1}
I20260806 22:44:39.362277 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:39.361472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:52425)
0806 22:44:39.361979 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:39.362062 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":275,"threads_started":1}
I20260806 22:44:39.426146 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:39.425335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:54633)
0806 22:44:39.425879 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:39.425978 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":331,"threads_started":1}
I20260806 22:44:39.902025 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:39.901117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:39409)
0806 22:44:39.901744 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:39.901840 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":299,"threads_started":1}
I20260806 22:44:40.364287 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:40.363321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:60799)
0806 22:44:40.363986 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:40.364103 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":349,"threads_started":1}
I20260806 22:44:40.427857 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:40.427121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:49023)
0806 22:44:40.427591 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:40.427688 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260806 22:44:40.904018 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:40.903147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:45005)
0806 22:44:40.903750 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:40.903850 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":341,"threads_started":1}
I20260806 22:44:41.366389 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:41.365464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:49153)
0806 22:44:41.366092 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:41.366198 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":396,"threads_started":1}
I20260806 22:44:41.429711 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:41.428824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:56803)
0806 22:44:41.429323 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:41.429418 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":221,"threads_started":1}
I20260806 22:44:41.906316 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:41.905148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:44273)
0806 22:44:41.905991 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:41.906088 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":266,"threads_started":1}
I20260806 22:44:42.368357 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:42.367432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:35223)
0806 22:44:42.368024 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:42.368121 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":269,"threads_started":1}
I20260806 22:44:42.431629 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:42.430831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:38961)
0806 22:44:42.431358 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:42.431458 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":276,"threads_started":1}
I20260806 22:44:42.908047 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:42.907288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:36551)
0806 22:44:42.907774 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:42.907863 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":247,"threads_started":1}
I20260806 22:44:43.370231 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:43.369339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:44691)
0806 22:44:43.369963 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:43.370054 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":351,"threads_started":1}
I20260806 22:44:43.433380 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:43.432671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:36711)
0806 22:44:43.433127 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:43.433209 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":220,"threads_started":1}
I20260806 22:44:43.909829 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:43.909041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:60901)
0806 22:44:43.909510 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:43.909653 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":231,"threads_started":1}
I20260806 22:44:44.372115 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:44.371322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:60413)
0806 22:44:44.371828 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:44.371917 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":260,"threads_started":1}
I20260806 22:44:44.435164 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:44.434387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:44805)
0806 22:44:44.434892 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:44.434999 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":278,"threads_started":1}
I20260806 22:44:44.911504 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:44.910721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:36027)
0806 22:44:44.911261 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:44.911345 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":322,"threads_started":1}
I20260806 22:44:45.374002 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:45.373118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:59443)
0806 22:44:45.373711 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:45.373800 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":314,"threads_started":1}
I20260806 22:44:45.436847 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:45.436125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:47337)
0806 22:44:45.436568 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:45.436668 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260806 22:44:45.913801 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:45.912842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:51033)
0806 22:44:45.913474 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:45.913624 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":330,"threads_started":1}
I20260806 22:44:46.375892 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:46.375081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:47081)
0806 22:44:46.375608 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:46.375709 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":260,"threads_started":1}
I20260806 22:44:46.438604 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:46.437861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:36765)
0806 22:44:46.438342 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:46.438425 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260806 22:44:46.915467 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:46.914738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:55195)
0806 22:44:46.915214 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:46.915299 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":246,"threads_started":1}
I20260806 22:44:47.378108 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:47.377195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:51903)
0806 22:44:47.377829 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:47.377933 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":355,"threads_started":1}
I20260806 22:44:47.440421 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:47.439551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:53021)
0806 22:44:47.440096 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:47.440197 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":259,"threads_started":1}
I20260806 22:44:47.917392 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:47.916462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:43127)
0806 22:44:47.917111 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:47.917198 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":340,"threads_started":1}
I20260806 22:44:48.379958 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:48.379102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:54117)
0806 22:44:48.379665 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:48.379759 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":308,"threads_started":1}
I20260806 22:44:48.442534 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:48.441475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:38165)
0806 22:44:48.442087 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:48.442229 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":295,"threads_started":1}
I20260806 22:44:48.919353 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:48.918522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:42163)
0806 22:44:48.919070 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:48.919166 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":281,"threads_started":1}
I20260806 22:44:49.382004 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:49.380988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:45297)
0806 22:44:49.381656 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:49.381772 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":331,"threads_started":1}
I20260806 22:44:49.444792 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:49.443966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:56995)
0806 22:44:49.444446 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:49.444590 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":242,"threads_started":1}
I20260806 22:44:49.921696 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:49.920695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.130:48575)
0806 22:44:49.921305 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:49.921430 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":341,"threads_started":1}
I20260806 22:44:50.362495 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 14077
I20260806 22:44:50.380782 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:50.363855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.23.100.129:43837)
0806 22:44:50.364494 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:50.364508 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:50.364548 (+ 40us) server_negotiation.cc:407] Connection header received
0806 22:44:50.366225 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:50.366237 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:50.366669 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:50.366973 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:50.368323 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:50.369649 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:50.370997 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:50.371687 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:50.371835 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:50.374776 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:50.374784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:50.374800 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:50.374839 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:50.377806 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:50.378496 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:50.378502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:50.378508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:50.378670 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:50.379001 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:50.379004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:50.379006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:50.379516 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:50.379740 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:50.380114 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 22:44:50.380501 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":332,"threads_started":1}
I20260806 22:44:50.382431 14293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:50.384100 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:50.383158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.131:33043)
0806 22:44:50.383693 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:50.383842 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":276,"threads_started":1}
I20260806 22:44:50.468681 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:50.466843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:36257 (local address 127.23.100.129:35077)
0806 22:44:50.468081 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:50.468331 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1100,"thread_start_us":740,"threads_started":1}
I20260806 22:44:50.469717 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 14077
I20260806 22:44:50.498739 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 14306
I20260806 22:44:50.517683 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:50.499795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.23.100.130:36179)
0806 22:44:50.500331 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:50.500344 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:50.500380 (+ 36us) server_negotiation.cc:407] Connection header received
0806 22:44:50.502093 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:50.502103 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:50.502552 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:50.502842 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:50.504357 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:50.505721 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:50.507217 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:50.507899 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:50.508035 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:50.511044 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:50.511052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:50.511067 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:50.511104 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:50.514250 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:50.514943 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:50.514950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:50.514956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:50.515120 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:50.515577 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:50.515581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:50.515583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:50.516086 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:50.516322 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:50.516872 (+ 550us) server_negotiation.cc:299] Negotiation successful
0806 22:44:50.517317 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260806 22:44:50.519268 14494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:50.606499 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 14306
I20260806 22:44:50.630585 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 14505
I20260806 22:44:50.649935 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:50.631590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.23.100.131:42519)
0806 22:44:50.632201 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:50.632215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:50.632256 (+ 41us) server_negotiation.cc:407] Connection header received
0806 22:44:50.634020 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:50.634031 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:50.634477 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:50.634840 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:50.636407 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:50.637708 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:50.639050 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:50.639705 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:50.639940 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:50.642991 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:50.643000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:50.643016 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:50.643057 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:50.646181 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:50.647019 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:50.647028 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:50.647035 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:50.647216 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:50.647662 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:50.647669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:50.647672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:50.648245 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:50.648604 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:50.649092 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 22:44:50.649600 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":281,"threads_started":1}
I20260806 22:44:50.651503 14680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:50.738149 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 14505
2026-08-06T22:44:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25115 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:44:50 dist-test-slave-zr1t krb5kdc[14784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:44:50 dist-test-slave-zr1t krb5kdc[14784](info): set up 2 sockets
Aug 06 22:44:50 dist-test-slave-zr1t krb5kdc[14784](info): commencing operation
krb5kdc: starting...
W20260806 22:44:52.139873 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.354s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:44:52 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:44:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:44:52.461885 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:41351
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:41351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:52.817451 14800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:52.817924 14800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:52.818109 14800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:52.825803 14800 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 22:44:52.825920 14800 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 22:44:52.825986 14800 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 22:44:52.826062 14800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:44:52.826162 14800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:52.826277 14800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:52.826361 14800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:44:52.826438 14800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:44:52.836687 14800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:41351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:41351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:52.838044 14800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:52.839442 14800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:52.852116 14816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:52.852031 14812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:52.852193 14809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:52.852742 14800 server_base.cc:1034] running on GCE node
I20260806 22:44:52.853610 14800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:52.855070 14800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:52.856323 14800 hybrid_clock.cc:648] HybridClock initialized: now 1786056292856279 us; error 52 us; skew 500 ppm
I20260806 22:44:52.856726 14800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:52 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056292, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:52.863651 14800 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:44:52.865357 14800 webserver.cc:460] Webserver started at http://127.23.100.190:38585/ using document root <none> and password file <none>
I20260806 22:44:52.866142 14800 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:52.866233 14800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:52.866511 14800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:52.869468 14800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa"
format_stamp: "Formatted at 2026-08-06 22:44:52 on dist-test-slave-zr1t"
server_key: "b3b6ad66b24d57fc99c5dcfc6d6977ea"
server_key_iv: "7088cee892ecd1d2e40724128fef72f0"
server_key_version: "encryptionkey@0"
I20260806 22:44:52.870302 14800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa"
format_stamp: "Formatted at 2026-08-06 22:44:52 on dist-test-slave-zr1t"
server_key: "b3b6ad66b24d57fc99c5dcfc6d6977ea"
server_key_iv: "7088cee892ecd1d2e40724128fef72f0"
server_key_version: "encryptionkey@0"
I20260806 22:44:52.875752 14800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:44:52.879571 14830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:52.881098 14800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:44:52.881378 14800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa"
format_stamp: "Formatted at 2026-08-06 22:44:52 on dist-test-slave-zr1t"
server_key: "b3b6ad66b24d57fc99c5dcfc6d6977ea"
server_key_iv: "7088cee892ecd1d2e40724128fef72f0"
server_key_version: "encryptionkey@0"
I20260806 22:44:52.881705 14800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:52.897428 14800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:52.902850 14800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:52.903194 14800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:52.922554 14800 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:41351
I20260806 22:44:52.922561 14905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:41351 every 8 connection(s)
I20260806 22:44:52.924031 14800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:44:52.927861 14909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:52.934078 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 14800
I20260806 22:44:52.934321 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:44:52.935014 23954 external_mini_cluster.cc:1428] Setting key 999c874c98677dd6b3eff6d647435dc0
I20260806 22:44:52.938987 14909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Bootstrap starting.
I20260806 22:44:52.942166 14909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:52.943300 14909 log.cc:824] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:52.946578 14909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: No bootstrap required, opened a new log
Aug 06 22:44:52 dist-test-slave-zr1t krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:52.956161 14909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:44:52.956645 14909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:52.956771 14909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ab9ef1715c42fd98ebb5ee3044c0aa, State: Initialized, Role: FOLLOWER
I20260806 22:44:52.957423 14909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:44:52.957791 14909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:44:52.957957 14909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:44:52.958163 14909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:52.960476 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:52.937212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.23.100.190:41351)
0806 22:44:52.938040 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:52.938053 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:52.938093 (+ 40us) server_negotiation.cc:407] Connection header received
0806 22:44:52.939921 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:52.939944 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:52.940321 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:52.940735 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:52.942332 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:52.943740 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:52.945745 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:52.946445 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:52.946584 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:52.951704 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:52.951713 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:52.951733 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:52.951794 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:52.955379 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:52.956284 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:52.956294 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:52.956305 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:52.956510 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:52.957064 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:52.957069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:52.957071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:52.957993 (+ 922us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:52.958368 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:52.958844 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 22:44:52.959345 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":395,"threads_started":1}
I20260806 22:44:52.960959 14909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:44:52.961509 14909 leader_election.cc:302] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75ab9ef1715c42fd98ebb5ee3044c0aa; no voters:
I20260806 22:44:52.962275 14909 leader_election.cc:288] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:44:52.962405 14916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:44:52.964938 14909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:44:52.969411 14916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 1 LEADER]: Becoming Leader. State: Replica: 75ab9ef1715c42fd98ebb5ee3044c0aa, State: Running, Role: LEADER
I20260806 22:44:52.970141 14916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:44:52.973475 14919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } } }
I20260806 22:44:52.973721 14920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ab9ef1715c42fd98ebb5ee3044c0aa. Latest consensus state: current_term: 1 leader_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } } }
I20260806 22:44:52.973954 14919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: This master's current role is: LEADER
I20260806 22:44:52.974037 14920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: This master's current role is: LEADER
W20260806 22:44:52.978873 14936 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 22:44:52.979020 14936 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 22:44:52.979140 14939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:44:52.980959 14939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:44:52.988883 14939 catalog_manager.cc:1347] Generated new cluster ID: 05b2e37a4eab470fa73dac9d3e0d5149
I20260806 22:44:52.989018 14939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:44:53.003026 14939 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:44:53.005105 14939 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:44:53.016620 14939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Generated new TSK 0
I20260806 22:44:53.017822 14939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:44:53.101313 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41351
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:53.455081 14945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:53.455359 14945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:53.455523 14945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:53.462872 14945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:53.462994 14945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:53.463210 14945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:44:53.473809 14945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:53.475113 14945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:53.476501 14945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:53.490844 14951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:53.490886 14955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:53.490866 14950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:53.495100 14945 server_base.cc:1034] running on GCE node
I20260806 22:44:53.495764 14945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:53.496775 14945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:53.498005 14945 hybrid_clock.cc:648] HybridClock initialized: now 1786056293497951 us; error 49 us; skew 500 ppm
I20260806 22:44:53.498327 14945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:53 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056293, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:53.501964 14945 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:44:53.503568 14945 webserver.cc:460] Webserver started at http://127.23.100.129:34465/ using document root <none> and password file <none>
I20260806 22:44:53.504300 14945 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:53.504453 14945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:53.504772 14945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:53.507824 14945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "309dc77a5d764c12bcf5d8510884e942"
format_stamp: "Formatted at 2026-08-06 22:44:53 on dist-test-slave-zr1t"
server_key: "1822bc7516d6cca397eef2712e9022e0"
server_key_iv: "b14b132351f560cffc48abc7277350cb"
server_key_version: "encryptionkey@0"
I20260806 22:44:53.508617 14945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "309dc77a5d764c12bcf5d8510884e942"
format_stamp: "Formatted at 2026-08-06 22:44:53 on dist-test-slave-zr1t"
server_key: "1822bc7516d6cca397eef2712e9022e0"
server_key_iv: "b14b132351f560cffc48abc7277350cb"
server_key_version: "encryptionkey@0"
I20260806 22:44:53.514047 14945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260806 22:44:53.517932 14966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:53.519466 14945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:44:53.519670 14945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "309dc77a5d764c12bcf5d8510884e942"
format_stamp: "Formatted at 2026-08-06 22:44:53 on dist-test-slave-zr1t"
server_key: "1822bc7516d6cca397eef2712e9022e0"
server_key_iv: "b14b132351f560cffc48abc7277350cb"
server_key_version: "encryptionkey@0"
I20260806 22:44:53.519899 14945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:53.556245 14945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:53.560930 14945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:53.561249 14945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:53.562321 14945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:53.563759 14945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:53.563874 14945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:53.564026 14945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:53.564102 14945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:53.601717 14945 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:43677
I20260806 22:44:53.601805 15148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:43677 every 8 connection(s)
I20260806 22:44:53.603065 14945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:44:53.603941 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 14945
I20260806 22:44:53.604182 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:44:53.605024 23954 external_mini_cluster.cc:1428] Setting key 3208965f3cfce689bdc4d85b04ba08ca
Aug 06 22:44:53 dist-test-slave-zr1t krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056293, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:53.628223 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:53.606061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:56693 (local address 127.23.100.190:41351)
0806 22:44:53.606564 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:53.606568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:53.607779 (+ 1211us) server_negotiation.cc:407] Connection header received
0806 22:44:53.610898 (+ 3119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:53.610902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:53.610998 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:53.611098 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:53.613387 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:53.614302 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:53.616939 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:53.617391 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:53.617494 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:53.621974 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:53.621979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:53.621982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:53.622018 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:53.624924 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:53.625758 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:53.625764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:53.625768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:53.625902 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:53.626374 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:53.626378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:53.626380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:53.626666 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:53.626833 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:53.627603 (+ 770us) server_negotiation.cc:299] Negotiation successful
0806 22:44:53.627845 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":290,"threads_started":1}
I20260806 22:44:53.629258 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:53.606677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:56693)
0806 22:44:53.607586 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:53.607649 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:53.610528 (+ 2879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:53.611410 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:53.611424 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:53.612265 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:53.613123 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:53.613151 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:53.614481 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:53.614489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:53.616766 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:53.616785 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:53.617650 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:53.617661 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:53.617992 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:53.618798 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:53.618834 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:53.621723 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:53.625076 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:53.625085 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:53.625101 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:53.625631 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:53.626022 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:53.626028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:53.626031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:53.626239 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:53.626804 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:53.626812 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:53.627192 (+ 380us) client_negotiation.cc:769] Sending connection context
0806 22:44:53.627482 (+ 290us) client_negotiation.cc:240] Negotiation successful
0806 22:44:53.628064 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":378,"threads_started":1}
I20260806 22:44:53.630978 15149 heartbeater.cc:347] Connected to a master server at 127.23.100.190:41351
I20260806 22:44:53.631301 15149 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:53.632045 15149 heartbeater.cc:511] Master 127.23.100.190:41351 requested a full tablet report, sending...
I20260806 22:44:53.634486 14857 ts_manager.cc:194] Registered new tserver with Master: 309dc77a5d764c12bcf5d8510884e942 (127.23.100.129:43677)
I20260806 22:44:53.636826 14857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:56693
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:44:53.676884 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41351
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:54.035427 15158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:54.035735 15158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:54.035903 15158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:54.043594 15158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:54.043718 15158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:54.043915 15158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:44:54.054044 15158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:54.055441 15158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:54.056860 15158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:54.071026 15166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:54.071050 15167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:54.071707 15169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:54.072088 15158 server_base.cc:1034] running on GCE node
I20260806 22:44:54.072746 15158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:54.073853 15158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:54.075096 15158 hybrid_clock.cc:648] HybridClock initialized: now 1786056294075029 us; error 64 us; skew 500 ppm
I20260806 22:44:54.075423 15158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:54 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:54.078786 15158 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:44:54.080484 15158 webserver.cc:460] Webserver started at http://127.23.100.130:43407/ using document root <none> and password file <none>
I20260806 22:44:54.081214 15158 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:54.081378 15158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:54.081777 15158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:54.084859 15158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "5276492e4d4b463ab2d6ea3e515cbe85"
format_stamp: "Formatted at 2026-08-06 22:44:54 on dist-test-slave-zr1t"
server_key: "e3bf98e29c8c9358b30c5e106cfe1b5e"
server_key_iv: "100cc7d0a545939b0dcd3d63bc723c74"
server_key_version: "encryptionkey@0"
I20260806 22:44:54.085769 15158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "5276492e4d4b463ab2d6ea3e515cbe85"
format_stamp: "Formatted at 2026-08-06 22:44:54 on dist-test-slave-zr1t"
server_key: "e3bf98e29c8c9358b30c5e106cfe1b5e"
server_key_iv: "100cc7d0a545939b0dcd3d63bc723c74"
server_key_version: "encryptionkey@0"
I20260806 22:44:54.091428 15158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:44:54.095463 15179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:54.096961 15158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:44:54.097177 15158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "5276492e4d4b463ab2d6ea3e515cbe85"
format_stamp: "Formatted at 2026-08-06 22:44:54 on dist-test-slave-zr1t"
server_key: "e3bf98e29c8c9358b30c5e106cfe1b5e"
server_key_iv: "100cc7d0a545939b0dcd3d63bc723c74"
server_key_version: "encryptionkey@0"
I20260806 22:44:54.097405 15158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:54.119329 15158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:54.123804 15158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:54.124163 15158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:54.125257 15158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:54.126726 15158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:54.126842 15158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:54.127000 15158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:54.127075 15158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:54.163261 15158 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:37829
I20260806 22:44:54.163319 15347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:37829 every 8 connection(s)
I20260806 22:44:54.164645 15158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:44:54.171015 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 15158
I20260806 22:44:54.171226 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:44:54.171844 23954 external_mini_cluster.cc:1428] Setting key c995b2c8b6a6b9729926743a46d43174
Aug 06 22:44:54 dist-test-slave-zr1t krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:54.186887 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.167567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:44891 (local address 127.23.100.190:41351)
0806 22:44:54.168096 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:54.168104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:54.169275 (+ 1171us) server_negotiation.cc:407] Connection header received
0806 22:44:54.171591 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:54.171596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:54.171680 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:54.171781 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:54.173752 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.174539 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.176196 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.176630 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.176776 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.180575 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:54.180580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:54.180583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:54.180619 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:54.183490 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:54.184301 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:54.184305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:54.184308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:54.184438 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:54.184905 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:54.184909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:54.184911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:54.185194 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:54.185362 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:54.186256 (+ 894us) server_negotiation.cc:299] Negotiation successful
0806 22:44:54.186500 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":250,"threads_started":1}
I20260806 22:44:54.188040 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.168180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:44891)
0806 22:44:54.169044 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:54.169135 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:54.171251 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:54.172119 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:54.172134 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:54.172851 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:54.173486 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.173506 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.174714 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.174719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.176029 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.176042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.176896 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.176904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.177152 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.177967 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:54.177992 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:54.180293 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:54.183658 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:54.183668 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:54.183688 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:54.184154 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:54.184558 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:54.184564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:54.184567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:54.184773 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:54.185336 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:54.185345 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:54.185787 (+ 442us) client_negotiation.cc:769] Sending connection context
0806 22:44:54.186121 (+ 334us) client_negotiation.cc:240] Negotiation successful
0806 22:44:54.186755 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":325,"threads_started":1}
I20260806 22:44:54.189713 15348 heartbeater.cc:347] Connected to a master server at 127.23.100.190:41351
I20260806 22:44:54.190053 15348 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:54.190805 15348 heartbeater.cc:511] Master 127.23.100.190:41351 requested a full tablet report, sending...
I20260806 22:44:54.192483 14857 ts_manager.cc:194] Registered new tserver with Master: 5276492e4d4b463ab2d6ea3e515cbe85 (127.23.100.130:37829)
I20260806 22:44:54.193892 14857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:44891
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:44:54.243022 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41351
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:44:54.599640 15364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:44:54.599931 15364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:44:54.600026 15364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:44:54.610067 15364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:44:54.610190 15364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:44:54.610314 15364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:44:54.620208 15364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:44:54.621445 15364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:44:54.622838 15364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:44:54.636907 15375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:54.636999 15374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:44:54.636956 15377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:44:54.637951 15364 server_base.cc:1034] running on GCE node
I20260806 22:44:54.638652 15364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:44:54.639686 15364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:44:54.640162 15149 heartbeater.cc:503] Master 127.23.100.190:41351 was elected leader, sending a full tablet report...
I20260806 22:44:54.640954 15364 hybrid_clock.cc:648] HybridClock initialized: now 1786056294640897 us; error 42 us; skew 500 ppm
I20260806 22:44:54.641386 15364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:44:54 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:44:54.645046 15364 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:44:54.646924 15364 webserver.cc:460] Webserver started at http://127.23.100.131:37285/ using document root <none> and password file <none>
I20260806 22:44:54.647708 15364 fs_manager.cc:359] Metadata directory not provided
I20260806 22:44:54.647878 15364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:44:54.648223 15364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:44:54.651523 15364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "fa992f7a6a0247fca3bdbea65cd07a83"
format_stamp: "Formatted at 2026-08-06 22:44:54 on dist-test-slave-zr1t"
server_key: "3091213eb24c42838d7320ea09401dec"
server_key_iv: "909761de59c77ff999a9ae304ddf33ec"
server_key_version: "encryptionkey@0"
I20260806 22:44:54.652401 15364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "fa992f7a6a0247fca3bdbea65cd07a83"
format_stamp: "Formatted at 2026-08-06 22:44:54 on dist-test-slave-zr1t"
server_key: "3091213eb24c42838d7320ea09401dec"
server_key_iv: "909761de59c77ff999a9ae304ddf33ec"
server_key_version: "encryptionkey@0"
I20260806 22:44:54.658407 15364 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 22:44:54.662412 15387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:54.663949 15364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:44:54.664167 15364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "fa992f7a6a0247fca3bdbea65cd07a83"
format_stamp: "Formatted at 2026-08-06 22:44:54 on dist-test-slave-zr1t"
server_key: "3091213eb24c42838d7320ea09401dec"
server_key_iv: "909761de59c77ff999a9ae304ddf33ec"
server_key_version: "encryptionkey@0"
I20260806 22:44:54.664408 15364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:44:54.677148 15364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:44:54.682129 15364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:44:54.682475 15364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:44:54.683548 15364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:44:54.684998 15364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:44:54.685133 15364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:54.685242 15364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:44:54.685290 15364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:44:54.721071 15364 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:46631
I20260806 22:44:54.721119 15544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:46631 every 8 connection(s)
I20260806 22:44:54.722443 15364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:44:54.726678 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 15364
I20260806 22:44:54.726910 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:44:54.727469 23954 external_mini_cluster.cc:1428] Setting key 1abb0b14986668a9a7590ac0236a37c6
Aug 06 22:44:54 dist-test-slave-zr1t krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:44:54.746065 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.725220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:59703 (local address 127.23.100.190:41351)
0806 22:44:54.726096 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:54.726101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:54.727294 (+ 1193us) server_negotiation.cc:407] Connection header received
0806 22:44:54.730151 (+ 2857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:54.730157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:54.730301 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:54.730435 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:54.732346 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.733215 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.735118 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.735544 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.735657 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.739512 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:54.739518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:54.739521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:54.739557 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:54.742770 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:54.743717 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:54.743721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:54.743724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:54.743853 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:54.744405 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:54.744408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:54.744410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:54.744691 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:54.744817 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:54.745582 (+ 765us) server_negotiation.cc:299] Negotiation successful
0806 22:44:54.745775 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":511,"threads_started":1}
I20260806 22:44:54.747053 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.725909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:59703)
0806 22:44:54.727079 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:54.727144 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:54.729437 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:54.730697 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:54.730706 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:54.731483 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:44:54.732133 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.732153 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.733500 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.733505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.734909 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.734930 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.735797 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.735803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.736022 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.736820 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:44:54.736845 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:44:54.739274 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:44:54.742928 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:54.742940 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:54.742966 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:54.743559 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:54.744024 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:44:54.744028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:44:54.744031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:44:54.744289 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:44:54.744804 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:44:54.744810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:44:54.745149 (+ 339us) client_negotiation.cc:769] Sending connection context
0806 22:44:54.745420 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 22:44:54.745913 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":521,"threads_started":1}
I20260806 22:44:54.748499 15545 heartbeater.cc:347] Connected to a master server at 127.23.100.190:41351
I20260806 22:44:54.748826 15545 heartbeater.cc:464] Registering TS with master...
I20260806 22:44:54.749647 15545 heartbeater.cc:511] Master 127.23.100.190:41351 requested a full tablet report, sending...
I20260806 22:44:54.751405 14857 ts_manager.cc:194] Registered new tserver with Master: fa992f7a6a0247fca3bdbea65cd07a83 (127.23.100.131:46631)
I20260806 22:44:54.752238 14857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:59703
I20260806 22:44:54.756687 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:44:54.775985 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.761673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.23.100.190:41351)
0806 22:44:54.761849 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:54.761854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:54.762488 (+ 634us) server_negotiation.cc:407] Connection header received
0806 22:44:54.762689 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:54.762694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:54.762808 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:54.762909 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:54.764472 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.765217 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.766804 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.767225 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.767364 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.770355 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:54.770360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:54.770363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:54.770396 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:54.773091 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:54.773909 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:54.773913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:54.773915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:54.774055 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:54.774448 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:54.774458 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:54.774460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:54.774788 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:54.775105 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:54.775410 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 22:44:54.775651 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 22:44:54.780301 14857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:44:54.787628 14857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:44:54.812757 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.802271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.23.100.129:43677)
0806 22:44:54.803537 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:54.803549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:54.804248 (+ 699us) server_negotiation.cc:407] Connection header received
0806 22:44:54.804461 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:54.804471 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:54.804742 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:54.805385 (+ 643us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:54.806158 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.807863 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.810151 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.811552 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.811767 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.811957 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:54.812221 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 22:44:54.812383 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":1978,"threads_started":1}
I20260806 22:44:54.812881 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.801824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43677 (local address 127.0.0.1:43246)
0806 22:44:54.804126 (+ 2302us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:54.804144 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:54.804271 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:54.805341 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:54.805348 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:54.805382 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:54.805946 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.805962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.808015 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.808022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.809996 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.810008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.811764 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.811771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.811958 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.812023 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 22:44:54.812155 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 22:44:54.812363 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2188,"mutex_wait_us":30,"thread_start_us":2026,"threads_started":1}
I20260806 22:44:54.814072 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.801391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:46631 (local address 127.0.0.1:48414)
0806 22:44:54.802474 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:54.802535 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:54.802752 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:54.805639 (+ 2887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:54.805643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:54.805673 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:54.806194 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.806210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.808422 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.808427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.810943 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.810962 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.812928 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.812935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.813203 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.813403 (+ 200us) client_negotiation.cc:769] Sending connection context
0806 22:44:54.813491 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 22:44:54.813657 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":230,"thread_start_us":1122,"threads_started":1}
I20260806 22:44:54.814100 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.804648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:37829 (local address 127.0.0.1:33894)
0806 22:44:54.805287 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:54.805302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:54.805417 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:54.806484 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:54.806491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:54.806519 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:54.806967 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.806979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.808719 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.808727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.810992 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.811005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.812943 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.812949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.813201 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.813349 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 22:44:54.813437 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 22:44:54.813588 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":109,"spinlock_wait_cycles":13056,"thread_start_us":361,"threads_started":1}
I20260806 22:44:54.814231 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.801559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.23.100.131:46631)
0806 22:44:54.805000 (+ 3441us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:54.805010 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:54.805042 (+ 32us) server_negotiation.cc:407] Connection header received
0806 22:44:54.805134 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:54.805141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:54.805380 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:54.805984 (+ 604us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:54.806851 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.808282 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.811159 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.812625 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.813431 (+ 806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.813619 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:54.813698 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 22:44:54.813854 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3262,"thread_start_us":3070,"threads_started":1}
I20260806 22:44:54.815219 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.805114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.23.100.130:37829)
0806 22:44:54.805938 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:54.805945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:54.805972 (+ 27us) server_negotiation.cc:407] Connection header received
0806 22:44:54.806067 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:54.806073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:54.806291 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:54.806498 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:54.807155 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.808574 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.811157 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.812625 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.814222 (+ 1597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.814638 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:54.814701 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 22:44:54.814879 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":525,"threads_started":1}
I20260806 22:44:54.817269 15040 tablet_service.cc:1467] Processing CreateTablet for tablet 6509ce943842473faf97a9742a48a126 (DEFAULT_TABLE table=test-table [id=19c2fcfbf914460e956366f7bdc1832e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:54.818651 15040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6509ce943842473faf97a9742a48a126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:54.818274 15454 tablet_service.cc:1467] Processing CreateTablet for tablet 6509ce943842473faf97a9742a48a126 (DEFAULT_TABLE table=test-table [id=19c2fcfbf914460e956366f7bdc1832e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:54.819032 15254 tablet_service.cc:1467] Processing CreateTablet for tablet 6509ce943842473faf97a9742a48a126 (DEFAULT_TABLE table=test-table [id=19c2fcfbf914460e956366f7bdc1832e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:44:54.819732 15454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6509ce943842473faf97a9742a48a126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:54.820346 15254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6509ce943842473faf97a9742a48a126. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:44:54.830514 15571 tablet_bootstrap.cc:492] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: Bootstrap starting.
I20260806 22:44:54.835036 15571 tablet_bootstrap.cc:654] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:54.836202 15571 log.cc:824] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:54.838863 15573 tablet_bootstrap.cc:492] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: Bootstrap starting.
I20260806 22:44:54.839496 15571 tablet_bootstrap.cc:492] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: No bootstrap required, opened a new log
I20260806 22:44:54.839804 15571 ts_tablet_manager.cc:1397] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260806 22:44:54.842271 15573 tablet_bootstrap.cc:654] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:54.843448 15572 tablet_bootstrap.cc:492] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: Bootstrap starting.
I20260806 22:44:54.843851 15573 log.cc:824] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:54.847254 15572 tablet_bootstrap.cc:654] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: Neither blocks nor log segments found. Creating new log.
I20260806 22:44:54.847579 15573 tablet_bootstrap.cc:492] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: No bootstrap required, opened a new log
I20260806 22:44:54.847916 15573 ts_tablet_manager.cc:1397] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 22:44:54.848783 15572 log.cc:824] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: Log is configured to *not* fsync() on all Append() calls
I20260806 22:44:54.848842 15571 raft_consensus.cc:348] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } } peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } }
I20260806 22:44:54.849303 15571 raft_consensus.cc:374] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:54.849435 15571 raft_consensus.cc:729] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309dc77a5d764c12bcf5d8510884e942, State: Initialized, Role: FOLLOWER
I20260806 22:44:54.850157 15571 consensus_queue.cc:260] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } } peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } }
I20260806 22:44:54.852077 15571 ts_tablet_manager.cc:1428] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260806 22:44:54.855579 15572 tablet_bootstrap.cc:492] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: No bootstrap required, opened a new log
W20260806 22:44:54.855744 15150 tablet.cc:2369] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:44:54.855969 15572 ts_tablet_manager.cc:1397] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260806 22:44:54.859210 15573 raft_consensus.cc:348] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } } peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } }
I20260806 22:44:54.859627 15573 raft_consensus.cc:374] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:54.859759 15573 raft_consensus.cc:729] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5276492e4d4b463ab2d6ea3e515cbe85, State: Initialized, Role: FOLLOWER
I20260806 22:44:54.860399 15573 consensus_queue.cc:260] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } } peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } }
I20260806 22:44:54.861589 15348 heartbeater.cc:503] Master 127.23.100.190:41351 was elected leader, sending a full tablet report...
I20260806 22:44:54.862195 15573 ts_tablet_manager.cc:1428] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260806 22:44:54.864753 15572 raft_consensus.cc:348] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } } peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } }
I20260806 22:44:54.865293 15572 raft_consensus.cc:374] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:44:54.865419 15572 raft_consensus.cc:729] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa992f7a6a0247fca3bdbea65cd07a83, State: Initialized, Role: FOLLOWER
I20260806 22:44:54.866115 15572 consensus_queue.cc:260] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } } peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } }
I20260806 22:44:54.867631 15545 heartbeater.cc:503] Master 127.23.100.190:41351 was elected leader, sending a full tablet report...
I20260806 22:44:54.868273 15572 ts_tablet_manager.cc:1428] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260806 22:44:54.916996 15349 tablet.cc:2369] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:44:54.938016 15579 raft_consensus.cc:482] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:44:54.938301 15579 raft_consensus.cc:504] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } } peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } }
I20260806 22:44:54.939903 15579 leader_election.cc:288] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa992f7a6a0247fca3bdbea65cd07a83 (127.23.100.131:46631), 309dc77a5d764c12bcf5d8510884e942 (127.23.100.129:43677)
I20260806 22:44:54.946784 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.940458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:39233 (local address 127.23.100.129:43677)
0806 22:44:54.940684 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:54.940690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:54.941331 (+ 641us) server_negotiation.cc:407] Connection header received
0806 22:44:54.941600 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:54.941606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:54.941669 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:54.941769 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:54.942377 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.943125 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.944978 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.945821 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.946018 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.946103 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:54.946393 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 22:44:54.946526 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 22:44:54.946928 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.940363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43677 (local address 127.23.100.130:39233)
0806 22:44:54.941195 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:54.941211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:54.941405 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:54.941792 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:54.941798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:54.941825 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:54.942203 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.942213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.943265 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.943272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.944790 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.944803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.946058 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.946064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.946222 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.946247 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 22:44:54.946355 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 22:44:54.946504 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":1,"thread_start_us":501,"threads_started":1}
I20260806 22:44:54.947949 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.940539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:50423 (local address 127.23.100.131:46631)
0806 22:44:54.940771 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:54.940779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:54.941978 (+ 1199us) server_negotiation.cc:407] Connection header received
0806 22:44:54.942255 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:54.942268 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:54.942343 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:54.942455 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:44:54.943079 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.943904 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.945943 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.946838 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.947072 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.947161 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:54.947608 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 22:44:54.947748 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 22:44:54.948072 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:54.940925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:46631 (local address 127.23.100.130:50423)
0806 22:44:54.941847 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0806 22:44:54.941864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:44:54.942068 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:44:54.942477 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:44:54.942482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:44:54.942508 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:44:54.942901 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.942912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.944074 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.944080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.945741 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:44:54.945756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:44:54.947222 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:44:54.947228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:44:54.947407 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:54.947440 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 22:44:54.947535 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 22:44:54.947665 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":3,"thread_start_us":502,"threads_started":1}
I20260806 22:44:54.948243 15071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6509ce943842473faf97a9742a48a126" candidate_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309dc77a5d764c12bcf5d8510884e942" is_pre_election: true
I20260806 22:44:54.948930 15071 raft_consensus.cc:2454] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5276492e4d4b463ab2d6ea3e515cbe85 in term 0.
I20260806 22:44:54.949252 15482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6509ce943842473faf97a9742a48a126" candidate_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" is_pre_election: true
I20260806 22:44:54.949748 15185 leader_election.cc:302] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309dc77a5d764c12bcf5d8510884e942, 5276492e4d4b463ab2d6ea3e515cbe85; no voters:
I20260806 22:44:54.949958 15482 raft_consensus.cc:2454] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5276492e4d4b463ab2d6ea3e515cbe85 in term 0.
I20260806 22:44:54.950264 15579 raft_consensus.cc:2781] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:44:54.950434 15579 raft_consensus.cc:482] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:44:54.950624 15579 raft_consensus.cc:3037] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:54.953037 15579 raft_consensus.cc:504] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } } peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } }
I20260806 22:44:54.953755 15579 leader_election.cc:288] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [CANDIDATE]: Term 1 election: Requested vote from peers fa992f7a6a0247fca3bdbea65cd07a83 (127.23.100.131:46631), 309dc77a5d764c12bcf5d8510884e942 (127.23.100.129:43677)
I20260806 22:44:54.954290 15482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6509ce943842473faf97a9742a48a126" candidate_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa992f7a6a0247fca3bdbea65cd07a83"
I20260806 22:44:54.954311 15071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6509ce943842473faf97a9742a48a126" candidate_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309dc77a5d764c12bcf5d8510884e942"
I20260806 22:44:54.954531 15071 raft_consensus.cc:3037] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:54.954516 15482 raft_consensus.cc:3037] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:44:54.956777 15071 raft_consensus.cc:2454] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5276492e4d4b463ab2d6ea3e515cbe85 in term 1.
I20260806 22:44:54.956823 15482 raft_consensus.cc:2454] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5276492e4d4b463ab2d6ea3e515cbe85 in term 1.
I20260806 22:44:54.957361 15181 leader_election.cc:302] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5276492e4d4b463ab2d6ea3e515cbe85, fa992f7a6a0247fca3bdbea65cd07a83; no voters:
I20260806 22:44:54.957784 15579 raft_consensus.cc:2781] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:44:54.958325 15579 raft_consensus.cc:686] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 1 LEADER]: Becoming Leader. State: Replica: 5276492e4d4b463ab2d6ea3e515cbe85, State: Running, Role: LEADER
I20260806 22:44:54.958907 15579 consensus_queue.cc:237] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } } peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } }
I20260806 22:44:54.963894 14857 catalog_manager.cc:5495] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5276492e4d4b463ab2d6ea3e515cbe85 (127.23.100.130). New cstate: current_term: 1 leader_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } health_report { overall_health: UNKNOWN } } }
W20260806 22:44:54.974246 15546 tablet.cc:2369] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:44:55.009171 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:55.003323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.23.100.130:37829)
0806 22:44:55.003633 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:55.003642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:55.003667 (+ 25us) server_negotiation.cc:407] Connection header received
0806 22:44:55.003886 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:55.003891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:55.003984 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:55.004118 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:44:55.004497 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:55.005267 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:55.006791 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:55.007256 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:55.007415 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:55.007927 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:44:55.008480 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:44:55.008601 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:55.008715 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 22:44:55.008811 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 22:44:55.017642 15071 raft_consensus.cc:1261] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942 [term 1 FOLLOWER]: Refusing update from remote peer 5276492e4d4b463ab2d6ea3e515cbe85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:44:55.017642 15482 raft_consensus.cc:1261] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83 [term 1 FOLLOWER]: Refusing update from remote peer 5276492e4d4b463ab2d6ea3e515cbe85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:44:55.018688 15579 consensus_queue.cc:1035] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:44:55.018983 15583 consensus_queue.cc:1035] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:44:55.032266 15587 mvcc.cc:205] Tried to move back new op lower bound from 7315686584382021632 to 7315686584155844608. Current Snapshot: MvccSnapshot[applied={T|T < 7315686584382021632}]
I20260806 22:44:55.047585 14857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:44:55.047901 14857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:44:55.052816 14857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Sending DeleteTablet for 3 replicas of tablet 6509ce943842473faf97a9742a48a126
I20260806 22:44:55.054260 15454 tablet_service.cc:1514] Processing DeleteTablet for tablet 6509ce943842473faf97a9742a48a126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:55 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:48414
I20260806 22:44:55.054582 15040 tablet_service.cc:1514] Processing DeleteTablet for tablet 6509ce943842473faf97a9742a48a126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:55 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:43246
I20260806 22:44:55.055339 15254 tablet_service.cc:1514] Processing DeleteTablet for tablet 6509ce943842473faf97a9742a48a126 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:44:55 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:33894
I20260806 22:44:55.055598 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 14945
I20260806 22:44:55.065721 15604 tablet_replica.cc:330] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: stopping tablet replica
I20260806 22:44:55.066375 15604 raft_consensus.cc:2229] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:44:55.067097 15604 raft_consensus.cc:2258] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:55.069938 15604 ts_tablet_manager.cc:1905] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:55.079010 15604 ts_tablet_manager.cc:1918] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:55.079304 15604 log.cc:1196] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/6509ce943842473faf97a9742a48a126
I20260806 22:44:55.079977 15604 ts_tablet_manager.cc:1939] T 6509ce943842473faf97a9742a48a126 P fa992f7a6a0247fca3bdbea65cd07a83: Deleting consensus metadata
I20260806 22:44:55.081187 15608 tablet_replica.cc:330] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: stopping tablet replica
I20260806 22:44:55.081915 15608 raft_consensus.cc:2229] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:44:55.082340 14831 catalog_manager.cc:4841] TS fa992f7a6a0247fca3bdbea65cd07a83 (127.23.100.131:46631): tablet 6509ce943842473faf97a9742a48a126 (table test-table [id=19c2fcfbf914460e956366f7bdc1832e]) successfully deleted
I20260806 22:44:55.082999 15605 tablet_replica.cc:330] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: stopping tablet replica
I20260806 22:44:55.083384 15605 raft_consensus.cc:2229] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:44:55.083858 15605 raft_consensus.cc:2258] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:55.084730 15608 raft_consensus.cc:2258] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:44:55.087404 15608 ts_tablet_manager.cc:1905] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:55.087596 15605 ts_tablet_manager.cc:1905] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:44:55.094633 15605 ts_tablet_manager.cc:1918] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:55.094919 15605 log.cc:1196] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/6509ce943842473faf97a9742a48a126
I20260806 22:44:55.095207 15608 ts_tablet_manager.cc:1918] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:44:55.095454 15608 log.cc:1196] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/6509ce943842473faf97a9742a48a126
I20260806 22:44:55.095675 15605 ts_tablet_manager.cc:1939] T 6509ce943842473faf97a9742a48a126 P 309dc77a5d764c12bcf5d8510884e942: Deleting consensus metadata
I20260806 22:44:55.096084 15608 ts_tablet_manager.cc:1939] T 6509ce943842473faf97a9742a48a126 P 5276492e4d4b463ab2d6ea3e515cbe85: Deleting consensus metadata
Aug 06 22:44:55 dist-test-slave-zr1t krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:44:55.097836 14835 catalog_manager.cc:4841] TS 309dc77a5d764c12bcf5d8510884e942 (127.23.100.129:43677): tablet 6509ce943842473faf97a9742a48a126 (table test-table [id=19c2fcfbf914460e956366f7bdc1832e]) successfully deleted
I20260806 22:44:55.098373 14831 catalog_manager.cc:4841] TS 5276492e4d4b463ab2d6ea3e515cbe85 (127.23.100.130:37829): tablet 6509ce943842473faf97a9742a48a126 (table test-table [id=19c2fcfbf914460e956366f7bdc1832e]) successfully deleted
I20260806 22:44:55.105994 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:55.056825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.23.100.129:43677)
0806 22:44:55.057031 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:55.057037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:55.085406 (+ 28369us) server_negotiation.cc:407] Connection header received
0806 22:44:55.085697 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:55.085702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:55.085810 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:55.085912 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:55.088516 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:55.089716 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:55.091849 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:55.092560 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:55.092707 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:55.098771 (+ 6064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:55.098787 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:55.098807 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:55.098865 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:55.102790 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:55.103612 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:55.103623 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:55.103627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:55.103804 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:55.104253 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:55.104258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:55.104262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:55.104753 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:55.105089 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:55.105279 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 22:44:55.105546 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 22:44:55.106726 15144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:55.210937 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 14945
I20260806 22:44:55.239886 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 15158
Aug 06 22:44:55 dist-test-slave-zr1t krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:44:55.260213 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:55.240964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.23.100.130:37829)
0806 22:44:55.241139 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:55.241143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:55.241161 (+ 18us) server_negotiation.cc:407] Connection header received
0806 22:44:55.241250 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:55.241254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:55.241352 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:55.241435 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:55.243238 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:55.244072 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:55.246247 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:55.246700 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:55.246801 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:55.252385 (+ 5584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:55.252405 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:55.252426 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:55.252471 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:55.256852 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:55.257754 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:55.257766 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:55.257769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:55.257969 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:55.258368 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:55.258373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:55.258377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:55.258957 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:55.259295 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:55.259570 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 22:44:55.259805 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 22:44:55.261176 15343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:55.377947 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 15158
I20260806 22:44:55.404567 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 15364
Aug 06 22:44:55 dist-test-slave-zr1t krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:44:55.421295 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:44:55.405479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.23.100.131:46631)
0806 22:44:55.405749 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 22:44:55.405762 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:44:55.405783 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:44:55.405991 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:44:55.405996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:44:55.406063 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:44:55.406148 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:44:55.407485 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:55.408199 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:55.409683 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:44:55.410091 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:44:55.410187 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:44:55.415064 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:44:55.415083 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:44:55.415101 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:44:55.415149 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:44:55.418276 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:55.419133 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:55.419142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:55.419144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:55.419316 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:44:55.419782 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:44:55.419787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:44:55.419789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:44:55.420186 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:44:55.420427 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 22:44:55.420726 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 22:44:55.420906 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260806 22:44:55.422118 15538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:55.515208 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 15364
I20260806 22:44:55.538144 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 14800
I20260806 22:44:55.539477 14900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:44:55.651758 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 14800
I20260806 22:45:15.682363 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:41351
--webserver_interface=127.23.100.190
--webserver_port=38585
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:41351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:45:16.033947 15616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:16.034343 15616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:16.034513 15616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:16.041795 15616 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 22:45:16.041944 15616 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 22:45:16.042040 15616 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 22:45:16.042140 15616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:45:16.042233 15616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:16.042377 15616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:16.042466 15616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:45:16.042549 15616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:45:16.052556 15616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:41351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:41351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=38585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:16.053996 15616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:16.055447 15616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:16.084244 15625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:16.084276 15623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:16.084308 15627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:16.085116 15616 server_base.cc:1034] running on GCE node
I20260806 22:45:16.086144 15616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:16.087806 15616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:16.089107 15616 hybrid_clock.cc:648] HybridClock initialized: now 1786056316089013 us; error 65 us; skew 500 ppm
I20260806 22:45:16.089658 15616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:16 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:16.096729 15616 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:45:16.098650 15616 webserver.cc:460] Webserver started at http://127.23.100.190:38585/ using document root <none> and password file <none>
I20260806 22:45:16.099545 15616 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:16.099711 15616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:16.106392 15616 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 22:45:16.109810 15634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:16.111305 15616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:45:16.111529 15616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa"
format_stamp: "Formatted at 2026-08-06 22:44:52 on dist-test-slave-zr1t"
server_key: "b3b6ad66b24d57fc99c5dcfc6d6977ea"
server_key_iv: "7088cee892ecd1d2e40724128fef72f0"
server_key_version: "encryptionkey@0"
I20260806 22:45:16.112663 15616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:16.132169 15616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:16.136832 15616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:16.137193 15616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:16.157640 15616 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:41351
I20260806 22:45:16.157661 15711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:41351 every 8 connection(s)
I20260806 22:45:16.159456 15616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:45:16.166220 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 15616
I20260806 22:45:16.167407 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:43677
--local_ip_for_outbound_sockets=127.23.100.129
--tserver_master_addrs=127.23.100.190:41351
--webserver_port=34465
--webserver_interface=127.23.100.129
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:45:16.167943 15713 sys_catalog.cc:263] Verifying existing consensus state
I20260806 22:45:16.173859 15713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Bootstrap starting.
I20260806 22:45:16.199293 15713 log.cc:824] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Log is configured to *not* fsync() on all Append() calls
I20260806 22:45:16.212373 15713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 22:45:16.212984 15713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Bootstrap complete.
I20260806 22:45:16.224087 15713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:45:16.224982 15713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ab9ef1715c42fd98ebb5ee3044c0aa, State: Initialized, Role: FOLLOWER
I20260806 22:45:16.225853 15713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:45:16.226123 15713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:45:16.226275 15713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:45:16.226495 15713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 1 FOLLOWER]: Advancing to term 2
I20260806 22:45:16.230688 15713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:45:16.231256 15713 leader_election.cc:302] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75ab9ef1715c42fd98ebb5ee3044c0aa; no voters:
I20260806 22:45:16.231953 15713 leader_election.cc:288] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 22:45:16.232105 15719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 2 FOLLOWER]: Leader election won for term 2
I20260806 22:45:16.233579 15719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 2 LEADER]: Becoming Leader. State: Replica: 75ab9ef1715c42fd98ebb5ee3044c0aa, State: Running, Role: LEADER
I20260806 22:45:16.234284 15719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:45:16.234530 15713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:45:16.237134 15723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ab9ef1715c42fd98ebb5ee3044c0aa. Latest consensus state: current_term: 2 leader_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } } }
I20260806 22:45:16.237627 15723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: This master's current role is: LEADER
I20260806 22:45:16.238224 15720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } } }
I20260806 22:45:16.238549 15720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: This master's current role is: LEADER
I20260806 22:45:16.243265 15731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:45:16.253929 15731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:45:16.255998 15731 catalog_manager.cc:1259] Loaded cluster ID: 05b2e37a4eab470fa73dac9d3e0d5149
I20260806 22:45:16.256199 15731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:45:16.262499 15731 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:45:16.264796 15731 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Loaded TSK: 0
I20260806 22:45:16.314456 15731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Generated new TSK 1
I20260806 22:45:16.314855 15731 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 22:45:16.567176 15716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:16.567531 15716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:16.567701 15716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:16.575289 15716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:16.575465 15716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:16.575646 15716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:45:16.586066 15716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:43677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=34465
--tserver_master_addrs=127.23.100.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:16.587494 15716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:16.588876 15716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:16.602843 15753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:16.602874 15754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:16.603596 15756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:16.604292 15716 server_base.cc:1034] running on GCE node
I20260806 22:45:16.605027 15716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:16.606297 15716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:16.607744 15716 hybrid_clock.cc:648] HybridClock initialized: now 1786056316607686 us; error 51 us; skew 500 ppm
I20260806 22:45:16.608126 15716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:16 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:16.612154 15716 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:45:16.613961 15716 webserver.cc:460] Webserver started at http://127.23.100.129:34465/ using document root <none> and password file <none>
I20260806 22:45:16.614827 15716 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:16.614979 15716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:16.622373 15716 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 22:45:16.625916 15772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:16.627660 15716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:45:16.627846 15716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "309dc77a5d764c12bcf5d8510884e942"
format_stamp: "Formatted at 2026-08-06 22:44:53 on dist-test-slave-zr1t"
server_key: "1822bc7516d6cca397eef2712e9022e0"
server_key_iv: "b14b132351f560cffc48abc7277350cb"
server_key_version: "encryptionkey@0"
I20260806 22:45:16.629189 15716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:16.657339 15716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:16.662643 15716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:16.662949 15716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:16.663996 15716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:45:16.665428 15716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:45:16.665583 15716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:16.665761 15716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:45:16.665854 15716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:16.698202 15716 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:43677
I20260806 22:45:16.698235 15968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:43677 every 8 connection(s)
I20260806 22:45:16.699724 15716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:45:16.708920 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 15716
I20260806 22:45:16.709887 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:37829
--local_ip_for_outbound_sockets=127.23.100.130
--tserver_master_addrs=127.23.100.190:41351
--webserver_port=43407
--webserver_interface=127.23.100.130
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:45:16 dist-test-slave-zr1t krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056316, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:45:16.727033 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:16.702855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:57401)
0806 22:45:16.703711 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:16.703756 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:16.705488 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:16.706933 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:16.706941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:16.707522 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:45:16.708045 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:16.708061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:16.710770 (+ 2709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:16.710774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:16.712000 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:16.712013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:16.713485 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:16.713490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:16.713753 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:16.714461 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:45:16.714490 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:45:16.716905 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:45:16.721988 (+ 5083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:45:16.721996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:45:16.722008 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:45:16.722493 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:45:16.723047 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:45:16.723051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:45:16.723053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:45:16.723248 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:45:16.724389 (+ 1141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:45:16.724395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:45:16.724713 (+ 318us) client_negotiation.cc:769] Sending connection context
0806 22:45:16.725024 (+ 311us) client_negotiation.cc:240] Negotiation successful
0806 22:45:16.725667 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":368,"threads_started":1}
I20260806 22:45:16.727034 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:16.702740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:57401 (local address 127.23.100.190:41351)
0806 22:45:16.703331 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:16.703342 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:16.703913 (+ 571us) server_negotiation.cc:407] Connection header received
0806 22:45:16.706101 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:16.706113 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:16.706478 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:16.706777 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:16.708378 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:16.710567 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:16.712157 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:16.713228 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:16.713447 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:16.717277 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:16.717291 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:16.717315 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:16.717377 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:16.721799 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:16.722647 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:16.722659 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:16.722669 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:16.722899 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:16.723420 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:16.723427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:16.723433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:16.724187 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:16.724410 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:16.725257 (+ 847us) server_negotiation.cc:299] Negotiation successful
0806 22:45:16.725689 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":258,"threads_started":1}
I20260806 22:45:16.729974 15969 heartbeater.cc:347] Connected to a master server at 127.23.100.190:41351
I20260806 22:45:16.730297 15969 heartbeater.cc:464] Registering TS with master...
I20260806 22:45:16.731046 15969 heartbeater.cc:511] Master 127.23.100.190:41351 requested a full tablet report, sending...
I20260806 22:45:16.733260 15657 ts_manager.cc:194] Registered new tserver with Master: 309dc77a5d764c12bcf5d8510884e942 (127.23.100.129:43677)
I20260806 22:45:16.735199 15657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:57401
W20260806 22:45:17.075560 15975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:17.075903 15975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:17.076083 15975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:17.083555 15975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:17.083716 15975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:17.083899 15975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:45:17.094029 15975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:37829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=43407
--tserver_master_addrs=127.23.100.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:17.095371 15975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:17.096760 15975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:17.110098 15980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:17.110178 15984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:17.110162 15981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:17.111459 15975 server_base.cc:1034] running on GCE node
I20260806 22:45:17.112118 15975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:17.113119 15975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:17.114353 15975 hybrid_clock.cc:648] HybridClock initialized: now 1786056317114306 us; error 57 us; skew 500 ppm
I20260806 22:45:17.114704 15975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:17 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:17.121979 15975 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:45:17.123687 15975 webserver.cc:460] Webserver started at http://127.23.100.130:43407/ using document root <none> and password file <none>
I20260806 22:45:17.124447 15975 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:17.124624 15975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:17.130266 15975 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 22:45:17.132997 15997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:17.134505 15975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:45:17.134681 15975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "5276492e4d4b463ab2d6ea3e515cbe85"
format_stamp: "Formatted at 2026-08-06 22:44:54 on dist-test-slave-zr1t"
server_key: "e3bf98e29c8c9358b30c5e106cfe1b5e"
server_key_iv: "100cc7d0a545939b0dcd3d63bc723c74"
server_key_version: "encryptionkey@0"
I20260806 22:45:17.135819 15975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:17.159188 15975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:17.164702 15975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:17.165009 15975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:17.166251 15975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:45:17.168025 15975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:45:17.168169 15975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:17.168284 15975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:45:17.168370 15975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:17.218602 15975 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:37829
I20260806 22:45:17.218655 16167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:37829 every 8 connection(s)
I20260806 22:45:17.220881 15975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:45:17.228109 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 15975
I20260806 22:45:17.229038 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:46631
--local_ip_for_outbound_sockets=127.23.100.131
--tserver_master_addrs=127.23.100.190:41351
--webserver_port=37285
--webserver_interface=127.23.100.131
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 22:45:17 dist-test-slave-zr1t krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:45:17.249991 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.225849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:58939 (local address 127.23.100.190:41351)
0806 22:45:17.226067 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.226075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.227073 (+ 998us) server_negotiation.cc:407] Connection header received
0806 22:45:17.230077 (+ 3004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.230085 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.230206 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.230328 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:17.232804 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.233973 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.236591 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.237450 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.237637 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.243341 (+ 5704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:17.243346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:17.243350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:17.243385 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:17.246418 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:17.247382 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:17.247388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:17.247392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:17.247530 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:17.248025 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:17.248031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:17.248034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:17.248320 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:17.248482 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.249334 (+ 852us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.249602 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 22:45:17.251151 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.226000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:58939)
0806 22:45:17.226878 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:17.226944 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:17.229700 (+ 2756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:17.230672 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:17.230686 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:17.231602 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:45:17.232551 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.232581 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.234197 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.234205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.236414 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.236435 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.238181 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.238191 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.238543 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.239523 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:45:17.239595 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:45:17.243076 (+ 3481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:45:17.246583 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:45:17.246592 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:45:17.246613 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:45:17.247271 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:45:17.247658 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:45:17.247662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:45:17.247666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:45:17.247918 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:45:17.248499 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:45:17.248509 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:45:17.248953 (+ 444us) client_negotiation.cc:769] Sending connection context
0806 22:45:17.249254 (+ 301us) client_negotiation.cc:240] Negotiation successful
0806 22:45:17.249853 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":325,"threads_started":1}
I20260806 22:45:17.252902 16168 heartbeater.cc:347] Connected to a master server at 127.23.100.190:41351
I20260806 22:45:17.253230 16168 heartbeater.cc:464] Registering TS with master...
I20260806 22:45:17.254060 16168 heartbeater.cc:511] Master 127.23.100.190:41351 requested a full tablet report, sending...
I20260806 22:45:17.255654 15657 ts_manager.cc:194] Registered new tserver with Master: 5276492e4d4b463ab2d6ea3e515cbe85 (127.23.100.130:37829)
I20260806 22:45:17.256495 15657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:58939
W20260806 22:45:17.586144 16173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:17.586520 16173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:17.586704 16173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:17.594201 16173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:17.594367 16173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:17.594617 16173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:45:17.605315 16173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:46631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=37285
--tserver_master_addrs=127.23.100.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:17.606698 16173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:17.608054 16173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:17.621941 16181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:17.621872 16183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:17.621899 16185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:17.622751 16173 server_base.cc:1034] running on GCE node
I20260806 22:45:17.623369 16173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:17.624311 16173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:17.625617 16173 hybrid_clock.cc:648] HybridClock initialized: now 1786056317625569 us; error 56 us; skew 500 ppm
I20260806 22:45:17.625960 16173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:17 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:17.629101 16173 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:45:17.630707 16173 webserver.cc:460] Webserver started at http://127.23.100.131:37285/ using document root <none> and password file <none>
I20260806 22:45:17.631424 16173 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:17.631594 16173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:17.637415 16173 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 22:45:17.640380 16195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:17.641791 16173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 22:45:17.641986 16173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "fa992f7a6a0247fca3bdbea65cd07a83"
format_stamp: "Formatted at 2026-08-06 22:44:54 on dist-test-slave-zr1t"
server_key: "3091213eb24c42838d7320ea09401dec"
server_key_iv: "909761de59c77ff999a9ae304ddf33ec"
server_key_version: "encryptionkey@0"
I20260806 22:45:17.643018 16173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:17.668087 16173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:17.672394 16173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:17.672690 16173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:17.673859 16173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:45:17.675256 16173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:45:17.675396 16173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:17.675521 16173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:45:17.675611 16173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:17.711134 16173 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:46631
I20260806 22:45:17.711221 16369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:46631 every 8 connection(s)
I20260806 22:45:17.712594 16173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:45:17.716869 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 16173
Aug 06 22:45:17 dist-test-slave-zr1t krb5kdc[14784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:45:17.735656 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.717933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.23.100.190:41351)
0806 22:45:17.718425 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.718429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.720028 (+ 1599us) server_negotiation.cc:407] Connection header received
0806 22:45:17.720164 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.720172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.720246 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.720355 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:17.721971 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.722944 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.724889 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.725631 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.725719 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.729065 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:17.729072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:17.729078 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:17.729118 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:17.732286 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:17.733233 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:17.733237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:17.733239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:17.733444 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:17.733888 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:17.733895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:17.733901 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:17.734300 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:17.734584 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.734948 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.735184 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":285,"threads_started":1}
I20260806 22:45:17.736874 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.715125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:47091 (local address 127.23.100.190:41351)
0806 22:45:17.715310 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.715315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.716528 (+ 1213us) server_negotiation.cc:407] Connection header received
0806 22:45:17.719417 (+ 2889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.719421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.719487 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.719603 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:17.721379 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.722425 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.724615 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.725167 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.725292 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.729833 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:17.729842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:17.729847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:17.729897 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:17.733261 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:17.734229 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:17.734233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:17.734235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:17.734435 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:17.734941 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:17.734946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:17.734949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:17.735350 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:17.735512 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.736360 (+ 848us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.736616 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 22:45:17.738051 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.715583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:47091)
0806 22:45:17.716331 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:17.716389 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:17.719055 (+ 2666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:17.719855 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:17.719862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:17.720636 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:45:17.721206 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.721224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.722594 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.722598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.724471 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.724490 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.725375 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.725380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.725651 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.726468 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:45:17.726503 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:45:17.729588 (+ 3085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:45:17.733436 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:45:17.733445 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:45:17.733461 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:45:17.734075 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:45:17.734558 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:45:17.734563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:45:17.734566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:45:17.734801 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:45:17.735552 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:45:17.735561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:45:17.736007 (+ 446us) client_negotiation.cc:769] Sending connection context
0806 22:45:17.736239 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 22:45:17.736806 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":273,"threads_started":1}
I20260806 22:45:17.738962 15969 heartbeater.cc:503] Master 127.23.100.190:41351 was elected leader, sending a full tablet report...
I20260806 22:45:17.740207 16370 heartbeater.cc:347] Connected to a master server at 127.23.100.190:41351
I20260806 22:45:17.740622 16370 heartbeater.cc:464] Registering TS with master...
I20260806 22:45:17.741334 16370 heartbeater.cc:511] Master 127.23.100.190:41351 requested a full tablet report, sending...
I20260806 22:45:17.742745 15656 ts_manager.cc:194] Registered new tserver with Master: fa992f7a6a0247fca3bdbea65cd07a83 (127.23.100.131:46631)
I20260806 22:45:17.743597 15656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:47091
I20260806 22:45:17.752107 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:45:17.758608 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.753055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.23.100.190:41351)
0806 22:45:17.753218 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.753223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.753781 (+ 558us) server_negotiation.cc:407] Connection header received
0806 22:45:17.753944 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.753954 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.754095 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.754220 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:45:17.754621 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.755354 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.756747 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.757160 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.757251 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.757815 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:45:17.758084 (+ 269us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 22:45:17.758423 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52884: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 22:45:17.758749 15972 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52884: token expired
W20260806 22:45:17.759343 15551 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.100.190:41351)
I20260806 22:45:17.772414 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.759917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.23.100.190:41351)
0806 22:45:17.760068 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.760071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.760086 (+ 15us) server_negotiation.cc:407] Connection header received
0806 22:45:17.760137 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.760140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.760203 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.760271 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:17.761750 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.762496 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.763877 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.764261 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.764382 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.767390 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:17.767396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:17.767399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:17.767430 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:17.769969 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:17.770638 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:17.770642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:17.770645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:17.770756 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:17.771145 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:17.771149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:17.771151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:17.771476 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:17.771672 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.772000 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.772180 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 22:45:17.776057 15656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:45:17.783465 15656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:45:17.816541 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.795717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:46631 (local address 127.0.0.1:35370)
0806 22:45:17.796523 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:17.796568 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:17.796729 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:17.808484 (+ 11755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:17.808498 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:17.808566 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:45:17.809248 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.809265 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.811672 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.811683 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.813717 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.813730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.815317 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.815323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.815642 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.815694 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 22:45:17.815843 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 22:45:17.816047 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":54,"thread_start_us":481,"threads_started":1}
I20260806 22:45:17.816514 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.796236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.23.100.131:46631)
0806 22:45:17.807158 (+ 10922us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.807172 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.807226 (+ 54us) server_negotiation.cc:407] Connection header received
0806 22:45:17.807395 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.807408 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.807770 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.809395 (+ 1625us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:45:17.809811 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.811497 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.813929 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.815060 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.815307 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.815576 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.815925 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.816090 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10625,"thread_start_us":11785,"threads_started":1}
I20260806 22:45:17.818938 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.795715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43677 (local address 127.0.0.1:44406)
0806 22:45:17.808894 (+ 13179us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:17.808927 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:17.809117 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:17.811287 (+ 2170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:17.811294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:17.811318 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:45:17.811895 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.811908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.813840 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.813847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.816149 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.816164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.818343 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.818347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.818463 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.818486 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 22:45:17.818560 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 22:45:17.818648 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12943,"mutex_wait_us":354,"thread_start_us":13696,"threads_started":1}
I20260806 22:45:17.819168 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.796035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.23.100.129:43677)
0806 22:45:17.810726 (+ 14691us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.810735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.810765 (+ 30us) server_negotiation.cc:407] Connection header received
0806 22:45:17.810860 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.810868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.811085 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.811265 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:45:17.812103 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.813649 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.816343 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.817901 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.818110 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.818278 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.818654 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.818816 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14504,"thread_start_us":14978,"threads_started":1}
I20260806 22:45:17.821390 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.797253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.23.100.130:37829)
0806 22:45:17.808062 (+ 10809us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.808073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.810235 (+ 2162us) server_negotiation.cc:407] Connection header received
0806 22:45:17.810357 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.810375 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.810639 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.810840 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:45:17.811775 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.813351 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.817295 (+ 3944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.820307 (+ 3012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.820505 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.820680 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.820901 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.821060 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10579,"thread_start_us":10797,"threads_started":1}
I20260806 22:45:17.821611 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.796986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:37829 (local address 127.0.0.1:60822)
0806 22:45:17.810028 (+ 13042us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:17.810052 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:17.810184 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:17.811114 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:17.811119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:17.811148 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:45:17.811574 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.811592 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.813545 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.813554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.817104 (+ 3550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.817120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.820529 (+ 3409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.820535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.820714 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.820750 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 22:45:17.820854 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 22:45:17.820984 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12869,"mutex_wait_us":33,"thread_start_us":11521,"threads_started":1}
I20260806 22:45:17.821070 16270 tablet_service.cc:1467] Processing CreateTablet for tablet 217eb59151dc4f54ae4fece5878b8e42 (DEFAULT_TABLE table=test-table [id=c98ef0dd28694a4db0e75bea6582ae05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:45:17.823048 15851 tablet_service.cc:1467] Processing CreateTablet for tablet 217eb59151dc4f54ae4fece5878b8e42 (DEFAULT_TABLE table=test-table [id=c98ef0dd28694a4db0e75bea6582ae05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:45:17.823681 16270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217eb59151dc4f54ae4fece5878b8e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:45:17.824527 15851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217eb59151dc4f54ae4fece5878b8e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:45:17.825026 16079 tablet_service.cc:1467] Processing CreateTablet for tablet 217eb59151dc4f54ae4fece5878b8e42 (DEFAULT_TABLE table=test-table [id=c98ef0dd28694a4db0e75bea6582ae05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:45:17.826097 16079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217eb59151dc4f54ae4fece5878b8e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:45:17.838636 16396 tablet_bootstrap.cc:492] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85: Bootstrap starting.
I20260806 22:45:17.842376 16400 tablet_bootstrap.cc:492] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: Bootstrap starting.
I20260806 22:45:17.844201 16396 tablet_bootstrap.cc:654] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85: Neither blocks nor log segments found. Creating new log.
I20260806 22:45:17.845214 16397 tablet_bootstrap.cc:492] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: Bootstrap starting.
I20260806 22:45:17.845614 16396 log.cc:824] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85: Log is configured to *not* fsync() on all Append() calls
I20260806 22:45:17.846300 16400 tablet_bootstrap.cc:654] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: Neither blocks nor log segments found. Creating new log.
I20260806 22:45:17.847903 16400 log.cc:824] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: Log is configured to *not* fsync() on all Append() calls
I20260806 22:45:17.848605 16397 tablet_bootstrap.cc:654] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: Neither blocks nor log segments found. Creating new log.
I20260806 22:45:17.850005 16397 log.cc:824] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: Log is configured to *not* fsync() on all Append() calls
I20260806 22:45:17.851583 16400 tablet_bootstrap.cc:492] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: No bootstrap required, opened a new log
I20260806 22:45:17.851869 16400 ts_tablet_manager.cc:1397] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 22:45:17.852090 16396 tablet_bootstrap.cc:492] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85: No bootstrap required, opened a new log
I20260806 22:45:17.852394 16396 ts_tablet_manager.cc:1397] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260806 22:45:17.853256 16397 tablet_bootstrap.cc:492] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: No bootstrap required, opened a new log
I20260806 22:45:17.853605 16397 ts_tablet_manager.cc:1397] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260806 22:45:17.859509 16400 raft_consensus.cc:348] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } } peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } }
I20260806 22:45:17.860040 16400 raft_consensus.cc:374] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:45:17.860185 16400 raft_consensus.cc:729] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa992f7a6a0247fca3bdbea65cd07a83, State: Initialized, Role: FOLLOWER
I20260806 22:45:17.861003 16400 consensus_queue.cc:260] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } } peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } }
I20260806 22:45:17.862321 16370 heartbeater.cc:503] Master 127.23.100.190:41351 was elected leader, sending a full tablet report...
I20260806 22:45:17.862892 16400 ts_tablet_manager.cc:1428] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260806 22:45:17.863549 16396 raft_consensus.cc:348] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } } peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } }
I20260806 22:45:17.864126 16396 raft_consensus.cc:374] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:45:17.864251 16396 raft_consensus.cc:729] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5276492e4d4b463ab2d6ea3e515cbe85, State: Initialized, Role: FOLLOWER
I20260806 22:45:17.864872 16396 consensus_queue.cc:260] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } } peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } }
I20260806 22:45:17.865480 16397 raft_consensus.cc:348] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } } peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } }
I20260806 22:45:17.866019 16168 heartbeater.cc:503] Master 127.23.100.190:41351 was elected leader, sending a full tablet report...
I20260806 22:45:17.866029 16397 raft_consensus.cc:374] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:45:17.866138 16397 raft_consensus.cc:729] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309dc77a5d764c12bcf5d8510884e942, State: Initialized, Role: FOLLOWER
I20260806 22:45:17.866698 16397 consensus_queue.cc:260] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } } peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } }
I20260806 22:45:17.866850 16396 ts_tablet_manager.cc:1428] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260806 22:45:17.868296 16397 ts_tablet_manager.cc:1428] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: Time spent starting tablet: real 0.015s user 0.007s sys 0.005s
I20260806 22:45:17.874931 16408 raft_consensus.cc:482] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:45:17.875186 16408 raft_consensus.cc:504] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } } peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } }
I20260806 22:45:17.876754 16408 leader_election.cc:288] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa992f7a6a0247fca3bdbea65cd07a83 (127.23.100.131:46631), 5276492e4d4b463ab2d6ea3e515cbe85 (127.23.100.130:37829)
I20260806 22:45:17.879170 16407 raft_consensus.cc:482] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:45:17.879499 16407 raft_consensus.cc:504] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } } peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } }
I20260806 22:45:17.887497 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.878178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:34467 (local address 127.23.100.130:37829)
0806 22:45:17.878375 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.878381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.878729 (+ 348us) server_negotiation.cc:407] Connection header received
0806 22:45:17.878914 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.878919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.878996 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.879093 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:45:17.879680 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.880519 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.884018 (+ 3499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.885863 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.886025 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.886135 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.886908 (+ 773us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.887099 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 22:45:17.887727 16407 leader_election.cc:288] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 309dc77a5d764c12bcf5d8510884e942 (127.23.100.129:43677), fa992f7a6a0247fca3bdbea65cd07a83 (127.23.100.131:46631)
I20260806 22:45:17.888866 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.878036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:37829 (local address 127.23.100.129:34467)
0806 22:45:17.878614 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:17.878633 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:17.878741 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:17.879123 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:17.879127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:17.879152 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:45:17.879523 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.879532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.880999 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.881007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.883775 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.883798 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.886381 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.886390 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.886665 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.886717 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 22:45:17.888028 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0806 22:45:17.888245 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":29,"thread_start_us":343,"threads_started":1}
I20260806 22:45:17.892577 16105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb59151dc4f54ae4fece5878b8e42" candidate_uuid: "309dc77a5d764c12bcf5d8510884e942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" is_pre_election: true
I20260806 22:45:17.893476 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.877043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:49413 (local address 127.23.100.131:46631)
0806 22:45:17.877281 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.877290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.878064 (+ 774us) server_negotiation.cc:407] Connection header received
0806 22:45:17.878449 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.878455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.878536 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.878624 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:45:17.879328 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.880437 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.889157 (+ 8720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.891042 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.891250 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.891361 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.893072 (+ 1711us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.893210 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 22:45:17.893617 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.877063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:46631 (local address 127.23.100.129:49413)
0806 22:45:17.877955 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:17.877978 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:17.878185 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:17.878683 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:17.878690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:17.878717 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:45:17.879110 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.879122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.880614 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.880621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.888922 (+ 8301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.888946 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.892242 (+ 3296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.892251 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.892541 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.892596 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 22:45:17.893041 (+ 445us) client_negotiation.cc:240] Negotiation successful
0806 22:45:17.893189 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":43,"thread_start_us":461,"threads_started":1}
I20260806 22:45:17.893689 16105 raft_consensus.cc:2454] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 309dc77a5d764c12bcf5d8510884e942 in term 0.
I20260806 22:45:17.894629 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb59151dc4f54ae4fece5878b8e42" candidate_uuid: "309dc77a5d764c12bcf5d8510884e942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" is_pre_election: true
I20260806 22:45:17.894742 15773 leader_election.cc:302] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309dc77a5d764c12bcf5d8510884e942, 5276492e4d4b463ab2d6ea3e515cbe85; no voters:
I20260806 22:45:17.895217 16302 raft_consensus.cc:2454] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 309dc77a5d764c12bcf5d8510884e942 in term 0.
I20260806 22:45:17.896301 16408 raft_consensus.cc:2781] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:45:17.896543 16408 raft_consensus.cc:482] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:45:17.896724 16408 raft_consensus.cc:3037] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:45:17.900116 16408 raft_consensus.cc:504] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } } peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } }
I20260806 22:45:17.900906 16408 leader_election.cc:288] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [CANDIDATE]: Term 1 election: Requested vote from peers fa992f7a6a0247fca3bdbea65cd07a83 (127.23.100.131:46631), 5276492e4d4b463ab2d6ea3e515cbe85 (127.23.100.130:37829)
I20260806 22:45:17.901456 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb59151dc4f54ae4fece5878b8e42" candidate_uuid: "309dc77a5d764c12bcf5d8510884e942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa992f7a6a0247fca3bdbea65cd07a83"
I20260806 22:45:17.901719 16302 raft_consensus.cc:3037] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:45:17.902153 16105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb59151dc4f54ae4fece5878b8e42" candidate_uuid: "309dc77a5d764c12bcf5d8510884e942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5276492e4d4b463ab2d6ea3e515cbe85"
I20260806 22:45:17.902411 16105 raft_consensus.cc:3037] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:45:17.904019 16302 raft_consensus.cc:2454] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309dc77a5d764c12bcf5d8510884e942 in term 1.
I20260806 22:45:17.904505 15773 leader_election.cc:302] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309dc77a5d764c12bcf5d8510884e942, fa992f7a6a0247fca3bdbea65cd07a83; no voters:
I20260806 22:45:17.904825 16408 raft_consensus.cc:2781] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:45:17.905815 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.882219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:58487 (local address 127.23.100.129:43677)
0806 22:45:17.882434 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.882441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.895157 (+ 12716us) server_negotiation.cc:407] Connection header received
0806 22:45:17.895464 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.895471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.895552 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.895649 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:45:17.896473 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.897621 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.900307 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.901716 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.901863 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.901944 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.905374 (+ 3430us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.905561 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 22:45:17.906003 16105 raft_consensus.cc:2454] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 309dc77a5d764c12bcf5d8510884e942 in term 1.
I20260806 22:45:17.906643 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.881803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43677 (local address 127.23.100.130:58487)
0806 22:45:17.895029 (+ 13226us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:17.895055 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:17.895289 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:17.895807 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:17.895814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:17.895847 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:45:17.896282 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.896300 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.897821 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.897826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.900111 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.900129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.904975 (+ 4846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.904982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.905169 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.905208 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 22:45:17.906048 (+ 840us) client_negotiation.cc:240] Negotiation successful
0806 22:45:17.906195 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13017,"thread_start_us":18000,"threads_started":1}
I20260806 22:45:17.907124 16408 raft_consensus.cc:686] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 1 LEADER]: Becoming Leader. State: Replica: 309dc77a5d764c12bcf5d8510884e942, State: Running, Role: LEADER
I20260806 22:45:17.907263 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.882531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:34671 (local address 127.23.100.131:46631)
0806 22:45:17.893757 (+ 11226us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.893765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.899642 (+ 5877us) server_negotiation.cc:407] Connection header received
0806 22:45:17.899846 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.899851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.899927 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.900018 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:45:17.900797 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.902051 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.904689 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.905990 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.906145 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.906210 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.906802 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.906974 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11113,"thread_start_us":16402,"threads_started":1}
I20260806 22:45:17.907905 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.881964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:46631 (local address 127.23.100.130:34671)
0806 22:45:17.899517 (+ 17553us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:17.899536 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:17.899700 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:17.900148 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:17.900152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:17.900172 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:45:17.900612 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.900627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.902260 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.902266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.904521 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:17.904537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.906380 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.906388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:17.906581 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.906628 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 22:45:17.907433 (+ 805us) client_negotiation.cc:240] Negotiation successful
0806 22:45:17.907560 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17454,"thread_start_us":10951,"threads_started":1}
I20260806 22:45:17.907800 15891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb59151dc4f54ae4fece5878b8e42" candidate_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309dc77a5d764c12bcf5d8510884e942" is_pre_election: true
I20260806 22:45:17.908053 16408 consensus_queue.cc:237] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } } peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } }
I20260806 22:45:17.909579 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb59151dc4f54ae4fece5878b8e42" candidate_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" is_pre_election: true
I20260806 22:45:17.909902 16302 raft_consensus.cc:2379] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5276492e4d4b463ab2d6ea3e515cbe85 in current term 1: Already voted for candidate 309dc77a5d764c12bcf5d8510884e942 in this term.
I20260806 22:45:17.910610 15999 leader_election.cc:302] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5276492e4d4b463ab2d6ea3e515cbe85; no voters: 309dc77a5d764c12bcf5d8510884e942, fa992f7a6a0247fca3bdbea65cd07a83
I20260806 22:45:17.911016 16407 raft_consensus.cc:2726] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 22:45:17.913499 15657 catalog_manager.cc:5495] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 reported cstate change: term changed from 0 to 1, leader changed from <none> to 309dc77a5d764c12bcf5d8510884e942 (127.23.100.129). New cstate: current_term: 1 leader_uuid: "309dc77a5d764c12bcf5d8510884e942" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309dc77a5d764c12bcf5d8510884e942" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 43677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 } health_report { overall_health: UNKNOWN } } }
I20260806 22:45:17.947242 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.941687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.23.100.129:43677)
0806 22:45:17.941851 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.941856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.942026 (+ 170us) server_negotiation.cc:407] Connection header received
0806 22:45:17.942200 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.942204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.942306 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.942436 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:45:17.942787 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.943552 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.944987 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.945405 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.945509 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:17.946029 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:45:17.946610 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:45:17.946717 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:17.946831 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 22:45:17.946916 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 22:45:17.952183 15970 tablet.cc:2369] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:45:17.955888 16302 raft_consensus.cc:1261] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83 [term 1 FOLLOWER]: Refusing update from remote peer 309dc77a5d764c12bcf5d8510884e942: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:45:17.955921 16105 raft_consensus.cc:1261] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 1 FOLLOWER]: Refusing update from remote peer 309dc77a5d764c12bcf5d8510884e942: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:45:17.956796 16408 consensus_queue.cc:1035] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa992f7a6a0247fca3bdbea65cd07a83" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 46631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:45:17.957144 16417 consensus_queue.cc:1035] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5276492e4d4b463ab2d6ea3e515cbe85" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 22:45:17.964363 16371 tablet.cc:2369] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:45:17.968653 16422 mvcc.cc:205] Tried to move back new op lower bound from 7315686678336815104 to 7315686678156476416. Current Snapshot: MvccSnapshot[applied={T|T < 7315686678336815104}]
I20260806 22:45:17.970502 16424 mvcc.cc:205] Tried to move back new op lower bound from 7315686678336815104 to 7315686678156476416. Current Snapshot: MvccSnapshot[applied={T|T < 7315686678336815104}]
I20260806 22:45:17.981994 15657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:45:17.982306 15657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:45:17.986027 15657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Sending DeleteTablet for 3 replicas of tablet 217eb59151dc4f54ae4fece5878b8e42
I20260806 22:45:17.987190 15851 tablet_service.cc:1514] Processing DeleteTablet for tablet 217eb59151dc4f54ae4fece5878b8e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:45:17 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:44406
I20260806 22:45:17.987416 16079 tablet_service.cc:1514] Processing DeleteTablet for tablet 217eb59151dc4f54ae4fece5878b8e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:45:17 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:60822
I20260806 22:45:17.987447 16270 tablet_service.cc:1514] Processing DeleteTablet for tablet 217eb59151dc4f54ae4fece5878b8e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:45:17 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:35370
I20260806 22:45:17.987983 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 15716
I20260806 22:45:18.000129 16436 tablet_replica.cc:330] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: stopping tablet replica
I20260806 22:45:18.000712 16436 raft_consensus.cc:2229] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:45:18.001799 16436 raft_consensus.cc:2258] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:45:18.003145 16440 tablet_replica.cc:330] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: stopping tablet replica
I20260806 22:45:18.003654 16440 raft_consensus.cc:2229] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:45:18.004259 16440 raft_consensus.cc:2258] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:45:18.005021 16436 ts_tablet_manager.cc:1905] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:45:18.005685 16437 tablet_replica.cc:330] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85: stopping tablet replica
I20260806 22:45:18.006170 16437 raft_consensus.cc:2229] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:45:18.006507 16440 ts_tablet_manager.cc:1905] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:45:18.006716 16437 raft_consensus.cc:2258] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:45:18.010949 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:17.989000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.23.100.129:43677)
0806 22:45:17.989241 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:17.989247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:17.989420 (+ 173us) server_negotiation.cc:407] Connection header received
0806 22:45:17.989666 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:17.989673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:17.989761 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:17.989884 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:17.992108 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.993275 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.995860 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:17.996582 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:17.996728 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:18.000774 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:18.000787 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:18.000806 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:18.000858 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:18.005219 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:18.008056 (+ 2837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:18.008066 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:18.008070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:18.008238 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:18.008724 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:18.008729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:18.008734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:18.009264 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:18.009619 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:18.010295 (+ 676us) server_negotiation.cc:299] Negotiation successful
0806 22:45:18.010559 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 22:45:18.012662 15958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:45:18.013478 16440 ts_tablet_manager.cc:1918] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:45:18.015285 16437 ts_tablet_manager.cc:1905] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:45:18.019160 16440 log.cc:1196] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/217eb59151dc4f54ae4fece5878b8e42
I20260806 22:45:18.019804 16440 ts_tablet_manager.cc:1939] T 217eb59151dc4f54ae4fece5878b8e42 P fa992f7a6a0247fca3bdbea65cd07a83: Deleting consensus metadata
I20260806 22:45:18.021678 15635 catalog_manager.cc:4841] TS fa992f7a6a0247fca3bdbea65cd07a83 (127.23.100.131:46631): tablet 217eb59151dc4f54ae4fece5878b8e42 (table test-table [id=c98ef0dd28694a4db0e75bea6582ae05]) successfully deleted
I20260806 22:45:18.023554 16437 ts_tablet_manager.cc:1918] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:45:18.023743 16437 log.cc:1196] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/217eb59151dc4f54ae4fece5878b8e42
I20260806 22:45:18.024338 16437 ts_tablet_manager.cc:1939] T 217eb59151dc4f54ae4fece5878b8e42 P 5276492e4d4b463ab2d6ea3e515cbe85: Deleting consensus metadata
I20260806 22:45:18.026453 15635 catalog_manager.cc:4841] TS 5276492e4d4b463ab2d6ea3e515cbe85 (127.23.100.130:37829): tablet 217eb59151dc4f54ae4fece5878b8e42 (table test-table [id=c98ef0dd28694a4db0e75bea6582ae05]) successfully deleted
I20260806 22:45:18.114908 16436 ts_tablet_manager.cc:1918] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:45:18.115238 16436 log.cc:1196] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/217eb59151dc4f54ae4fece5878b8e42
I20260806 22:45:18.115760 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 15716
W20260806 22:45:18.143777 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.100.129:43677 (error 108)
I20260806 22:45:18.144043 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 15975
I20260806 22:45:18.145778 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:18.145345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43677 (local address 127.0.0.1:44426)
0806 22:45:18.145510 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:18.145636 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 22:45:18.146229 15637 catalog_manager.cc:4584] TS 309dc77a5d764c12bcf5d8510884e942 (127.23.100.129:43677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 217eb59151dc4f54ae4fece5878b8e42: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43677: connect: Connection refused (error 111)
I20260806 22:45:18.159875 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:18.145020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.23.100.130:37829)
0806 22:45:18.145228 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:18.145235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:18.145256 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:45:18.146031 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:18.146035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:18.146110 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:18.146194 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:18.147769 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:18.148556 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:18.150041 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:18.150465 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:18.150591 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:18.153326 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:18.153342 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:18.153359 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:18.153396 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:18.156532 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:18.157394 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:18.157402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:18.157404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:18.157630 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:18.158060 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:18.158063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:18.158066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:18.158568 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:18.158786 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:18.159233 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 22:45:18.159499 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 22:45:18.160579 16160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:45:18.200001 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:18.199492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43677 (local address 127.0.0.1:44430)
0806 22:45:18.199719 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:18.199832 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 22:45:18.201110 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:18.200736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43677 (local address 127.0.0.1:44434)
0806 22:45:18.200897 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:18.200988 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 22:45:18.262061 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:18.261473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43677 (local address 127.0.0.1:44436)
0806 22:45:18.261734 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:18.261849 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260806 22:45:18.262810 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 15975
I20260806 22:45:18.263305 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:18.262820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43677 (local address 127.0.0.1:44438)
0806 22:45:18.263079 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:18.263163 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260806 22:45:18.289973 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 16173
I20260806 22:45:18.304593 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:18.290803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.23.100.131:46631)
0806 22:45:18.291007 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:18.291011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:18.291047 (+ 36us) server_negotiation.cc:407] Connection header received
0806 22:45:18.291209 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:18.291213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:18.291293 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:18.291376 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:18.292853 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:18.293560 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:18.294986 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:18.295709 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:18.295863 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:18.298773 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:18.298788 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:18.298803 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:18.298855 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:18.301919 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:18.302670 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:18.302677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:18.302679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:18.302815 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:18.303176 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:18.303179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:18.303181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:18.303560 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:18.303784 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:18.304051 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 22:45:18.304244 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 22:45:18.305260 16363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:45:18.344113 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:18.343565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43677 (local address 127.0.0.1:44454)
0806 22:45:18.343801 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:18.343901 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 22:45:18.345249 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:18.344852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:43677 (local address 127.0.0.1:44468)
0806 22:45:18.345061 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:18.345143 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:43677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 22:45:18.397655 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 16173
I20260806 22:45:18.422500 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 15616
I20260806 22:45:18.423676 15704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:45:18.539960 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 15616
I20260806 22:45:38.572507 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:41351
--webserver_interface=127.23.100.190
--webserver_port=38585
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:41351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:45:38.942543 16451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:38.942854 16451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:38.942948 16451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:38.950376 16451 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 22:45:38.950484 16451 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 22:45:38.950524 16451 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 22:45:38.950572 16451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:45:38.950628 16451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:38.950700 16451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:38.950748 16451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:45:38.950798 16451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:45:38.961022 16451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:41351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:41351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=38585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:38.962330 16451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:38.963652 16451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:38.975656 16458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:38.975806 16459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:38.976241 16462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:38.977365 16451 server_base.cc:1034] running on GCE node
I20260806 22:45:38.978245 16451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:38.979786 16451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:38.981050 16451 hybrid_clock.cc:648] HybridClock initialized: now 1786056338980959 us; error 67 us; skew 500 ppm
I20260806 22:45:38.981460 16451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:38 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056338, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:38.985194 16451 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:45:38.986949 16451 webserver.cc:460] Webserver started at http://127.23.100.190:38585/ using document root <none> and password file <none>
I20260806 22:45:38.987766 16451 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:38.987931 16451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:38.994237 16451 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 22:45:38.997462 16469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:38.998991 16451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:45:38.999210 16451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa"
format_stamp: "Formatted at 2026-08-06 22:44:52 on dist-test-slave-zr1t"
server_key: "b3b6ad66b24d57fc99c5dcfc6d6977ea"
server_key_iv: "7088cee892ecd1d2e40724128fef72f0"
server_key_version: "encryptionkey@0"
I20260806 22:45:39.000358 16451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:39.023465 16451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:39.028151 16451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:39.028488 16451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:39.055855 16451 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:41351
I20260806 22:45:39.055896 16548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:41351 every 8 connection(s)
I20260806 22:45:39.057696 16451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:45:39.062274 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 16451
I20260806 22:45:39.063393 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:43677
--local_ip_for_outbound_sockets=127.23.100.129
--tserver_master_addrs=127.23.100.190:41351
--webserver_port=34465
--webserver_interface=127.23.100.129
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:45:39.066437 16549 sys_catalog.cc:263] Verifying existing consensus state
I20260806 22:45:39.071789 16549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Bootstrap starting.
I20260806 22:45:39.095323 16549 log.cc:824] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Log is configured to *not* fsync() on all Append() calls
I20260806 22:45:39.112960 16549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 22:45:39.113656 16549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Bootstrap complete.
I20260806 22:45:39.123306 16549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:45:39.124225 16549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ab9ef1715c42fd98ebb5ee3044c0aa, State: Initialized, Role: FOLLOWER
I20260806 22:45:39.124895 16549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:45:39.125152 16549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:45:39.125294 16549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:45:39.125486 16549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 2 FOLLOWER]: Advancing to term 3
I20260806 22:45:39.129354 16549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:45:39.129914 16549 leader_election.cc:302] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75ab9ef1715c42fd98ebb5ee3044c0aa; no voters:
I20260806 22:45:39.130643 16549 leader_election.cc:288] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 22:45:39.130806 16554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 3 FOLLOWER]: Leader election won for term 3
I20260806 22:45:39.131551 16554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [term 3 LEADER]: Becoming Leader. State: Replica: 75ab9ef1715c42fd98ebb5ee3044c0aa, State: Running, Role: LEADER
I20260806 22:45:39.132158 16554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } }
I20260806 22:45:39.132825 16549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:45:39.134649 16557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } } }
I20260806 22:45:39.135042 16557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: This master's current role is: LEADER
I20260806 22:45:39.135609 16555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ab9ef1715c42fd98ebb5ee3044c0aa. Latest consensus state: current_term: 3 leader_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ab9ef1715c42fd98ebb5ee3044c0aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41351 } } }
I20260806 22:45:39.135922 16555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa [sys.catalog]: This master's current role is: LEADER
I20260806 22:45:39.142838 16565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:45:39.153430 16565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:45:39.155584 16565 catalog_manager.cc:1259] Loaded cluster ID: 05b2e37a4eab470fa73dac9d3e0d5149
I20260806 22:45:39.155795 16565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:45:39.161440 16565 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:45:39.162904 16587 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 22:45:39.163623 16565 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Loaded TSK: 0
I20260806 22:45:39.163775 16565 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 75ab9ef1715c42fd98ebb5ee3044c0aa: Loaded TSK: 1
F20260806 22:45:39.164608 16565 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f20493209ad google::LogMessage::Fail() at ??:0
@ 0x7f204932388a google::LogMessage::SendToLog() at ??:0
@ 0x7f20493204de google::LogMessage::Flush() at ??:0
@ 0x7f2049321a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f204d0847d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f204d07373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f205ed4f1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f205ed4e783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f205edda2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f205edda0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f205ee245e9 std::function<>::operator()() at ??:0
@ 0x7f205ed52cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f205ed50c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f205edd8610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f205edd8439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f204bf6c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f204bf6f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f204bf6f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f204bf4aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f204f2246db start_thread at ??:0
@ 0x7f20481ba61f clone at ??:0
W20260806 22:45:39.489089 16551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:39.489403 16551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:39.489574 16551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:39.497318 16551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:39.497460 16551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:39.497671 16551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:45:39.508278 16551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:43677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=34465
--tserver_master_addrs=127.23.100.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:39.509685 16551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:39.511045 16551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:39.525044 16633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:39.525020 16635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:39.524987 16632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:39.526217 16551 server_base.cc:1034] running on GCE node
I20260806 22:45:39.526858 16551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:39.527874 16551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:39.529142 16551 hybrid_clock.cc:648] HybridClock initialized: now 1786056339529095 us; error 54 us; skew 500 ppm
I20260806 22:45:39.529497 16551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:39 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056339, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:39.532941 16551 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:45:39.534607 16551 webserver.cc:460] Webserver started at http://127.23.100.129:34465/ using document root <none> and password file <none>
I20260806 22:45:39.535337 16551 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:39.535504 16551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:39.541074 16551 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 22:45:39.543998 16646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:39.545436 16551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:45:39.545648 16551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "309dc77a5d764c12bcf5d8510884e942"
format_stamp: "Formatted at 2026-08-06 22:44:53 on dist-test-slave-zr1t"
server_key: "1822bc7516d6cca397eef2712e9022e0"
server_key_iv: "b14b132351f560cffc48abc7277350cb"
server_key_version: "encryptionkey@0"
I20260806 22:45:39.546666 16551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:39.562727 16551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:39.567066 16551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:39.567369 16551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:39.568451 16551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:45:39.570533 16659 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260806 22:45:39.576119 16659 ts_tablet_manager.cc:1852] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260806 22:45:39.577169 16659 ts_tablet_manager.cc:1905] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:45:39.584772 16659 ts_tablet_manager.cc:1918] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:45:39.585116 16659 ts_tablet_manager.cc:1939] T 217eb59151dc4f54ae4fece5878b8e42 P 309dc77a5d764c12bcf5d8510884e942: Deleting consensus metadata
I20260806 22:45:39.585628 16551 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260806 22:45:39.585764 16551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260806 22:45:39.585922 16551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:45:39.586017 16551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:39.623319 16551 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:43677
I20260806 22:45:39.623420 16836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:43677 every 8 connection(s)
I20260806 22:45:39.624706 16551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:45:39.629824 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 16551
I20260806 22:45:39.630607 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:39.627952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:50731)
0806 22:45:39.628752 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:39.629094 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":318,"threads_started":1}
I20260806 22:45:39.630827 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:37829
--local_ip_for_outbound_sockets=127.23.100.130
--tserver_master_addrs=127.23.100.190:41351
--webserver_port=43407
--webserver_interface=127.23.100.130
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:45:39.631990 16839 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:41351 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.190:41351: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
I20260806 22:45:39.633291 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:39.632737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:41695)
0806 22:45:39.632985 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:39.633101 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 22:45:39.634539 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:39.634023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:52733)
0806 22:45:39.634222 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:39.634341 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 22:45:39.634859 16839 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 22:45:39.998306 16843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:39.998660 16843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:39.998834 16843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:40.006256 16843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:40.006448 16843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:40.006628 16843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:45:40.016604 16843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:37829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=43407
--tserver_master_addrs=127.23.100.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:40.018078 16843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:40.019454 16843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:40.033236 16853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:40.033387 16857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:40.033354 16850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:40.034458 16843 server_base.cc:1034] running on GCE node
I20260806 22:45:40.035087 16843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:40.036074 16843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:40.037340 16843 hybrid_clock.cc:648] HybridClock initialized: now 1786056340037293 us; error 40 us; skew 500 ppm
I20260806 22:45:40.037737 16843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:40 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056340, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:40.040978 16843 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:45:40.042600 16843 webserver.cc:460] Webserver started at http://127.23.100.130:43407/ using document root <none> and password file <none>
I20260806 22:45:40.043361 16843 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:40.043496 16843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:40.049008 16843 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 22:45:40.051997 16867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:40.053383 16843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:45:40.053573 16843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "5276492e4d4b463ab2d6ea3e515cbe85"
format_stamp: "Formatted at 2026-08-06 22:44:54 on dist-test-slave-zr1t"
server_key: "e3bf98e29c8c9358b30c5e106cfe1b5e"
server_key_iv: "100cc7d0a545939b0dcd3d63bc723c74"
server_key_version: "encryptionkey@0"
I20260806 22:45:40.054571 16843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:40.072146 16843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:40.076432 16843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:40.076727 16843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:40.077800 16843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:45:40.079185 16843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:45:40.079275 16843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:40.079447 16843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:45:40.079552 16843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:40.114774 16843 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:37829
I20260806 22:45:40.114833 17052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:37829 every 8 connection(s)
I20260806 22:45:40.116271 16843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:45:40.121301 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:40.119374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:58857)
0806 22:45:40.120132 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:40.120378 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":304,"threads_started":1}
W20260806 22:45:40.122083 17054 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:41351 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.190:41351: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
I20260806 22:45:40.122974 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:40.122621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:46573)
0806 22:45:40.122784 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:40.122861 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 22:45:40.123818 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:40.123486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:52341)
0806 22:45:40.123614 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:40.123692 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 22:45:40.124212 17054 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 22:45:40.125980 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 16843
I20260806 22:45:40.126699 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:46631
--local_ip_for_outbound_sockets=127.23.100.131
--tserver_master_addrs=127.23.100.190:41351
--webserver_port=37285
--webserver_interface=127.23.100.131
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:45:40.479614 17058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:40.479930 17058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:40.480125 17058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:40.487764 17058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:40.487891 17058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:40.488155 17058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:45:40.498445 17058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:46631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=37285
--tserver_master_addrs=127.23.100.190:41351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:40.499840 17058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:40.501259 17058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:40.516086 17064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:40.516026 17067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:40.516047 17065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:40.517024 17058 server_base.cc:1034] running on GCE node
I20260806 22:45:40.517771 17058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:40.518850 17058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:40.520082 17058 hybrid_clock.cc:648] HybridClock initialized: now 1786056340520030 us; error 49 us; skew 500 ppm
I20260806 22:45:40.520416 17058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:40 dist-test-slave-zr1t krb5kdc[14784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056340, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:40.524052 17058 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:45:40.525820 17058 webserver.cc:460] Webserver started at http://127.23.100.131:37285/ using document root <none> and password file <none>
I20260806 22:45:40.526547 17058 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:40.526705 17058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:40.532378 17058 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260806 22:45:40.535688 17077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:40.537222 17058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 22:45:40.537462 17058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "fa992f7a6a0247fca3bdbea65cd07a83"
format_stamp: "Formatted at 2026-08-06 22:44:54 on dist-test-slave-zr1t"
server_key: "3091213eb24c42838d7320ea09401dec"
server_key_iv: "909761de59c77ff999a9ae304ddf33ec"
server_key_version: "encryptionkey@0"
I20260806 22:45:40.538571 17058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:40.561329 17058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:40.566340 17058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:40.566599 17058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:40.567636 17058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:45:40.569015 17058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:45:40.569111 17058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:40.569257 17058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:45:40.569350 17058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:40.605638 17058 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:46631
I20260806 22:45:40.605739 17257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:46631 every 8 connection(s)
I20260806 22:45:40.607172 17058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:45:40.610371 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 17058
I20260806 22:45:40.612632 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:40.610260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:45393)
0806 22:45:40.611183 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:40.611441 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":389,"threads_started":1}
W20260806 22:45:40.613261 17258 heartbeater.cc:644] Failed to heartbeat to 127.23.100.190:41351 (0 consecutive failures): Network error: Failed to ping master at 127.23.100.190:41351: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
I20260806 22:45:40.614167 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:40.613813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:53393)
0806 22:45:40.613982 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:40.614064 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 22:45:40.615016 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:40.614683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:57463)
0806 22:45:40.614825 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:40.614911 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 22:45:40.615249 17258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 22:45:40.636302 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:40.635496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:52647)
0806 22:45:40.635990 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:40.636073 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":234,"threads_started":1}
I20260806 22:45:41.125687 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:41.124770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:56719)
0806 22:45:41.125255 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:41.125373 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":249,"threads_started":1}
I20260806 22:45:41.616976 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:41.616051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:51861)
0806 22:45:41.616617 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:41.616725 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":344,"threads_started":1}
I20260806 22:45:41.638334 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:41.637279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:46761)
0806 22:45:41.637918 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:41.638079 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":368,"threads_started":1}
I20260806 22:45:42.127499 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:42.126709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:48641)
0806 22:45:42.127232 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:42.127320 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":274,"threads_started":1}
I20260806 22:45:42.618835 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:42.618031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:38937)
0806 22:45:42.618560 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:42.618651 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":288,"threads_started":1}
I20260806 22:45:42.640190 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:42.639381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:52545)
0806 22:45:42.639899 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:42.640002 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":251,"threads_started":1}
I20260806 22:45:43.129405 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:43.128654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:41557)
0806 22:45:43.129115 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:43.129202 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":243,"threads_started":1}
I20260806 22:45:43.620544 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:43.619712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:56361)
0806 22:45:43.620264 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:43.620348 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":268,"threads_started":1}
I20260806 22:45:43.642117 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:43.641205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:56563)
0806 22:45:43.641825 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:43.641924 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":339,"threads_started":1}
I20260806 22:45:44.131330 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:44.130420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:39009)
0806 22:45:44.131035 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:44.131125 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":283,"threads_started":1}
I20260806 22:45:44.622401 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:44.621508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:40551)
0806 22:45:44.622130 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:44.622212 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":311,"threads_started":1}
I20260806 22:45:44.643975 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:44.643141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:54427)
0806 22:45:44.643713 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:44.643803 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":302,"threads_started":1}
I20260806 22:45:45.133963 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:45.132568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:34191)
0806 22:45:45.133443 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:45.133698 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":485,"threads_started":1}
I20260806 22:45:45.625106 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:45.623804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:40787)
0806 22:45:45.624692 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:45.624825 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":465,"threads_started":1}
I20260806 22:45:45.646559 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:45.645217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:36833)
0806 22:45:45.646092 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:45.646240 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":460,"threads_started":1}
I20260806 22:45:46.136652 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:46.135385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:46901)
0806 22:45:46.136200 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:46.136369 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":435,"threads_started":1}
I20260806 22:45:46.626850 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:46.626158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:35397)
0806 22:45:46.626606 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:46.626701 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":233,"threads_started":1}
I20260806 22:45:46.648594 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:46.647678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:33439)
0806 22:45:46.648293 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:46.648377 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":338,"threads_started":1}
I20260806 22:45:47.138829 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:47.137939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:52623)
0806 22:45:47.138501 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:47.138592 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":263,"threads_started":1}
I20260806 22:45:47.628706 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:47.627811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:44991)
0806 22:45:47.628388 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:47.628482 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":245,"threads_started":1}
I20260806 22:45:47.650391 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:47.649588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:59879)
0806 22:45:47.650126 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:47.650209 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":324,"threads_started":1}
I20260806 22:45:48.140506 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:48.139803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:58353)
0806 22:45:48.140258 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:48.140341 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":229,"threads_started":1}
I20260806 22:45:48.630410 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:48.629699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:54087)
0806 22:45:48.630136 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:48.630224 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":217,"threads_started":1}
I20260806 22:45:48.651983 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:48.651254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:53947)
0806 22:45:48.651725 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:48.651815 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":248,"threads_started":1}
I20260806 22:45:49.142203 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:49.141346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:47589)
0806 22:45:49.141847 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:49.141927 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":260,"threads_started":1}
I20260806 22:45:49.632072 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:49.631275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:55319)
0806 22:45:49.631812 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:49.631897 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":272,"threads_started":1}
I20260806 22:45:49.653695 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:49.652947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:52093)
0806 22:45:49.653382 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:49.653465 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":235,"threads_started":1}
I20260806 22:45:50.143955 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:50.143158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:37691)
0806 22:45:50.143634 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:50.143770 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":262,"threads_started":1}
I20260806 22:45:50.633849 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:50.633041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:40523)
0806 22:45:50.633590 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:50.633681 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":250,"threads_started":1}
I20260806 22:45:50.655613 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:50.654668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:43095)
0806 22:45:50.655309 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:50.655392 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":443,"threads_started":1}
I20260806 22:45:51.145690 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:51.144831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:36403)
0806 22:45:51.145350 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:51.145447 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":288,"threads_started":1}
I20260806 22:45:51.635501 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:51.634766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:41273)
0806 22:45:51.635233 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:51.635332 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":256,"threads_started":1}
I20260806 22:45:51.657195 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:51.656523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:53997)
0806 22:45:51.656939 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:51.657028 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":221,"threads_started":1}
I20260806 22:45:52.147542 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:52.146677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:56619)
0806 22:45:52.147262 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:52.147384 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":321,"threads_started":1}
I20260806 22:45:52.637295 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:52.636589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:38865)
0806 22:45:52.637049 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:52.637138 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":242,"threads_started":1}
I20260806 22:45:52.659099 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:52.658197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:35791)
0806 22:45:52.658754 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:52.658893 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":292,"threads_started":1}
I20260806 22:45:53.149489 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:53.148493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:51903)
0806 22:45:53.149082 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:53.149224 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":253,"threads_started":1}
I20260806 22:45:53.639221 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:53.638363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:42133)
0806 22:45:53.638922 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:53.639030 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":280,"threads_started":1}
I20260806 22:45:53.660895 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:53.660104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:56973)
0806 22:45:53.660586 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:53.660675 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
I20260806 22:45:54.151389 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:54.150554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:41051)
0806 22:45:54.151027 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:54.151124 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":235,"threads_started":1}
I20260806 22:45:54.641062 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:54.640175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:43339)
0806 22:45:54.640691 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:54.640805 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":292,"threads_started":1}
I20260806 22:45:54.662974 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:54.662191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:33339)
0806 22:45:54.662682 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:54.662794 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":254,"threads_started":1}
I20260806 22:45:55.153584 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:55.152639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.130:51443)
0806 22:45:55.153206 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:55.153329 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":284,"threads_started":1}
I20260806 22:45:55.620118 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 16551
I20260806 22:45:55.643011 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:55.621606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.23.100.129:43677)
0806 22:45:55.622176 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:55.622191 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:55.622228 (+ 37us) server_negotiation.cc:407] Connection header received
0806 22:45:55.623939 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:55.623949 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:55.624366 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:55.624724 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:55.626271 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:55.628254 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:55.629511 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:55.630241 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:55.630385 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:55.636573 (+ 6188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:55.636582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:55.636599 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:55.636645 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:55.639574 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:55.640286 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:55.640293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:55.640300 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:55.640493 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:55.640809 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:55.640812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:55.640814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:55.641363 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:55.641736 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:55.642094 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 22:45:55.642623 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":279,"threads_started":1}
I20260806 22:45:55.643242 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:55.642395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.131:46273)
0806 22:45:55.642873 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:55.643002 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":239,"threads_started":1}
I20260806 22:45:55.644415 16830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:45:55.731832 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:55.730772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41351 (local address 127.23.100.129:51035)
0806 22:45:55.731360 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:55.731568 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.190:41351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":261,"threads_started":1}
I20260806 22:45:55.731972 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 16551
I20260806 22:45:55.757699 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 16843
I20260806 22:45:55.775967 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:55.758814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.23.100.130:37829)
0806 22:45:55.759397 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:55.759411 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:55.759465 (+ 54us) server_negotiation.cc:407] Connection header received
0806 22:45:55.761125 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:55.761134 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:55.761669 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:55.762026 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:55.763475 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:55.764686 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:55.766121 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:55.766778 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:55.766914 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:55.769587 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:55.769597 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:55.769612 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:55.769650 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:55.772598 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:55.773346 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:55.773353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:55.773360 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:55.773636 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:55.774061 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:55.774064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:55.774066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:55.774602 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:55.774832 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:55.775233 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 22:45:55.775680 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":281,"threads_started":1}
I20260806 22:45:55.777419 17042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:45:55.863265 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 16843
I20260806 22:45:55.890808 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 17058
I20260806 22:45:55.909803 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:55.891795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.23.100.131:46631)
0806 22:45:55.892370 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:55.892385 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:55.892428 (+ 43us) server_negotiation.cc:407] Connection header received
0806 22:45:55.894219 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:55.894230 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:55.894660 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:55.895039 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:55.896592 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:55.897931 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:55.899379 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:55.900048 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:55.900251 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:55.903140 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:55.903147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:55.903162 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:55.903210 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:55.906230 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:55.907020 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:55.907028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:55.907035 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:55.907208 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:55.907612 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:55.907617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:55.907620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:55.908154 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:55.908478 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:55.908989 (+ 511us) server_negotiation.cc:299] Negotiation successful
0806 22:45:55.909420 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":271,"threads_started":1}
I20260806 22:45:55.911283 17251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:45:55.996109 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 17058
2026-08-06T22:45:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65259 ms)
[----------] 30 tests from SecurityITest (309477 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:45:56 dist-test-slave-zr1t krb5kdc[17354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:45:56 dist-test-slave-zr1t krb5kdc[17354](info): set up 2 sockets
Aug 06 22:45:56 dist-test-slave-zr1t krb5kdc[17354](info): commencing operation
krb5kdc: starting...
W20260806 22:45:57.389115 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.344s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:45:57 dist-test-slave-zr1t krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:45:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:45:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:45:57.736562 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:46345
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:45:58.103197 17374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:58.103511 17374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:58.103665 17374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:58.111265 17374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:45:58.111390 17374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:58.111501 17374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:58.111610 17374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:45:58.111672 17374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:45:58.122442 17374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:46345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:58.123855 17374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:58.125224 17374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:58.138196 17385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:58.138150 17388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:58.138365 17384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:58.139659 17374 server_base.cc:1034] running on GCE node
I20260806 22:45:58.140588 17374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:58.142349 17374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:58.143635 17374 hybrid_clock.cc:648] HybridClock initialized: now 1786056358143573 us; error 54 us; skew 500 ppm
I20260806 22:45:58.144045 17374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:58 dist-test-slave-zr1t krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056358, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:58.147593 17374 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:45:58.149319 17374 webserver.cc:460] Webserver started at http://127.23.100.190:34899/ using document root <none> and password file <none>
I20260806 22:45:58.150120 17374 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:58.150274 17374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:58.150601 17374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:45:58.153796 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "098664a50b2b40fd9c3ed77924ec2259"
format_stamp: "Formatted at 2026-08-06 22:45:58 on dist-test-slave-zr1t"
server_key: "3b60747259255dbe19daf48940d5774b"
server_key_iv: "e5baaccafdf43c6124358bd12829e3b4"
server_key_version: "encryptionkey@0"
I20260806 22:45:58.154624 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "098664a50b2b40fd9c3ed77924ec2259"
format_stamp: "Formatted at 2026-08-06 22:45:58 on dist-test-slave-zr1t"
server_key: "3b60747259255dbe19daf48940d5774b"
server_key_iv: "e5baaccafdf43c6124358bd12829e3b4"
server_key_version: "encryptionkey@0"
I20260806 22:45:58.160465 17374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 22:45:58.164480 17398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:58.166317 17374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:45:58.166535 17374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "098664a50b2b40fd9c3ed77924ec2259"
format_stamp: "Formatted at 2026-08-06 22:45:58 on dist-test-slave-zr1t"
server_key: "3b60747259255dbe19daf48940d5774b"
server_key_iv: "e5baaccafdf43c6124358bd12829e3b4"
server_key_version: "encryptionkey@0"
I20260806 22:45:58.166778 17374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:58.190263 17374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:58.195451 17374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:58.195808 17374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:58.217037 17374 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:46345
I20260806 22:45:58.217070 17479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:46345 every 8 connection(s)
I20260806 22:45:58.218653 17374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:45:58.220367 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 17374
I20260806 22:45:58.220609 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:45:58.221642 23954 external_mini_cluster.cc:1428] Setting key 114a5e58730f779433f0dea36aff5d61
I20260806 22:45:58.224232 17480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 22:45:58 dist-test-slave-zr1t krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:45:58.240012 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259: Bootstrap starting.
I20260806 22:45:58.245188 17480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259: Neither blocks nor log segments found. Creating new log.
I20260806 22:45:58.246698 17480 log.cc:824] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259: Log is configured to *not* fsync() on all Append() calls
I20260806 22:45:58.250504 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259: No bootstrap required, opened a new log
I20260806 22:45:58.250891 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:58.224022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.23.100.190:46345)
0806 22:45:58.224938 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:58.224955 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:58.225008 (+ 53us) server_negotiation.cc:407] Connection header received
0806 22:45:58.227688 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:58.227719 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:58.228329 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:58.228868 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:58.230452 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:58.232815 (+ 2363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:58.234402 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:58.235616 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:58.235863 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:58.240755 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:58.240767 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:58.240789 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:58.240853 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:58.245749 (+ 4896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:58.246746 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:58.246756 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:58.246766 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:58.247017 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:58.247611 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:58.247619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:58.247623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:58.248329 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:58.248698 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:58.249187 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 22:45:58.249763 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":395,"threads_started":1}
I20260806 22:45:58.259083 17480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098664a50b2b40fd9c3ed77924ec2259" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46345 } }
I20260806 22:45:58.259553 17480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:45:58.259687 17480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098664a50b2b40fd9c3ed77924ec2259, State: Initialized, Role: FOLLOWER
I20260806 22:45:58.260349 17480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098664a50b2b40fd9c3ed77924ec2259" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46345 } }
I20260806 22:45:58.260622 17480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:45:58.260890 17480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:45:58.261157 17480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:45:58.264024 17480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098664a50b2b40fd9c3ed77924ec2259" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46345 } }
I20260806 22:45:58.264616 17480 leader_election.cc:302] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 098664a50b2b40fd9c3ed77924ec2259; no voters:
I20260806 22:45:58.265370 17480 leader_election.cc:288] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:45:58.265602 17486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:45:58.266911 17486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [term 1 LEADER]: Becoming Leader. State: Replica: 098664a50b2b40fd9c3ed77924ec2259, State: Running, Role: LEADER
I20260806 22:45:58.267498 17486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098664a50b2b40fd9c3ed77924ec2259" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46345 } }
I20260806 22:45:58.267928 17480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:45:58.270514 17487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "098664a50b2b40fd9c3ed77924ec2259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098664a50b2b40fd9c3ed77924ec2259" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46345 } } }
I20260806 22:45:58.271166 17487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [sys.catalog]: This master's current role is: LEADER
I20260806 22:45:58.270872 17489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 098664a50b2b40fd9c3ed77924ec2259. Latest consensus state: current_term: 1 leader_uuid: "098664a50b2b40fd9c3ed77924ec2259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098664a50b2b40fd9c3ed77924ec2259" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 46345 } } }
I20260806 22:45:58.271342 17489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259 [sys.catalog]: This master's current role is: LEADER
I20260806 22:45:58.273844 17500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:45:58.278884 17500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:45:58.289371 17500 catalog_manager.cc:1347] Generated new cluster ID: e910f495151b465e86c7209a8b9afb15
I20260806 22:45:58.289537 17500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:45:58.304098 17500 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:45:58.305744 17500 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:45:58.313585 17500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259: Generated new TSK 0
I20260806 22:45:58.314545 17500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:45:58.417024 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:46345
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:45:58.787072 17523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:58.787403 17523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:58.787561 17523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:58.795581 17523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:58.795739 17523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:58.795905 17523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:45:58.805815 17523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:46345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:58.807124 17523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:58.808415 17523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:58.823235 17531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:58.823267 17533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:58.823522 17536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:58.824203 17523 server_base.cc:1034] running on GCE node
I20260806 22:45:58.824887 17523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:58.825891 17523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:58.827100 17523 hybrid_clock.cc:648] HybridClock initialized: now 1786056358827061 us; error 28 us; skew 500 ppm
I20260806 22:45:58.827420 17523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:58 dist-test-slave-zr1t krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056358, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:58.830647 17523 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:45:58.832116 17523 webserver.cc:460] Webserver started at http://127.23.100.129:35331/ using document root <none> and password file <none>
I20260806 22:45:58.832805 17523 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:58.832954 17523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:58.833273 17523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:45:58.836324 17523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "82db840e9ef349c99fb633ff5f188c32"
format_stamp: "Formatted at 2026-08-06 22:45:58 on dist-test-slave-zr1t"
server_key: "aa19b494b7e7fde5e07fb34ec8b1a97a"
server_key_iv: "a3a130d49b680e4627780c4f80e21d06"
server_key_version: "encryptionkey@0"
I20260806 22:45:58.837100 17523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "82db840e9ef349c99fb633ff5f188c32"
format_stamp: "Formatted at 2026-08-06 22:45:58 on dist-test-slave-zr1t"
server_key: "aa19b494b7e7fde5e07fb34ec8b1a97a"
server_key_iv: "a3a130d49b680e4627780c4f80e21d06"
server_key_version: "encryptionkey@0"
I20260806 22:45:58.842195 17523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 22:45:58.845736 17544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:58.847002 17523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:45:58.847173 17523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "82db840e9ef349c99fb633ff5f188c32"
format_stamp: "Formatted at 2026-08-06 22:45:58 on dist-test-slave-zr1t"
server_key: "aa19b494b7e7fde5e07fb34ec8b1a97a"
server_key_iv: "a3a130d49b680e4627780c4f80e21d06"
server_key_version: "encryptionkey@0"
I20260806 22:45:58.847383 17523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:58.862232 17523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:58.866798 17523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:58.867105 17523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:58.868199 17523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:45:58.869912 17523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:45:58.870013 17523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:58.870162 17523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:45:58.870235 17523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:58.905689 17523 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:38073
I20260806 22:45:58.905706 17708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:38073 every 8 connection(s)
I20260806 22:45:58.907042 17523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:45:58.916632 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 17523
I20260806 22:45:58.916877 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:45:58.917757 23954 external_mini_cluster.cc:1428] Setting key 80339ebe9dcdd7cfca559964e29b8350
Aug 06 22:45:58 dist-test-slave-zr1t krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056358, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:45:58.929498 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:58.910204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:51945 (local address 127.23.100.190:46345)
0806 22:45:58.910652 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:58.910656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:58.911667 (+ 1011us) server_negotiation.cc:407] Connection header received
0806 22:45:58.913399 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:58.913404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:58.913489 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:58.913645 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:58.915128 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:58.915860 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:58.918345 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:58.918759 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:58.918864 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:58.922432 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:58.922438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:58.922441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:58.922476 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:58.925677 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:58.926678 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:58.926685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:58.926691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:58.926842 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:58.927359 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:58.927366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:58.927371 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:58.927707 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:58.927845 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:58.928807 (+ 962us) server_negotiation.cc:299] Negotiation successful
0806 22:45:58.929054 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":229,"threads_started":1}
I20260806 22:45:58.930811 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:58.910766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:46345 (local address 127.23.100.129:51945)
0806 22:45:58.911483 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:58.911521 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:58.913138 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:58.913857 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:58.913865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:58.914391 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:45:58.914942 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:58.914959 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:58.916012 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:58.916015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:58.918182 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:58.918198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:58.918999 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:58.919005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:58.919205 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:58.919941 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:45:58.919968 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:45:58.922189 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:45:58.925829 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:45:58.925840 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:45:58.925861 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:45:58.926551 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:45:58.926961 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:45:58.926968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:45:58.926974 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:45:58.927244 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:45:58.927843 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:45:58.927853 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:45:58.928356 (+ 503us) client_negotiation.cc:769] Sending connection context
0806 22:45:58.928704 (+ 348us) client_negotiation.cc:240] Negotiation successful
0806 22:45:58.929295 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":278,"threads_started":1}
I20260806 22:45:58.932407 17709 heartbeater.cc:347] Connected to a master server at 127.23.100.190:46345
I20260806 22:45:58.932734 17709 heartbeater.cc:464] Registering TS with master...
I20260806 22:45:58.933634 17709 heartbeater.cc:511] Master 127.23.100.190:46345 requested a full tablet report, sending...
I20260806 22:45:58.935837 17427 ts_manager.cc:194] Registered new tserver with Master: 82db840e9ef349c99fb633ff5f188c32 (127.23.100.129:38073)
I20260806 22:45:58.937299 17427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:51945
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:45:58.990718 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:46345
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:45:59.339464 17719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:59.339731 17719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:59.339892 17719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:59.347527 17719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:59.347635 17719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:59.347833 17719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:45:59.357898 17719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:46345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:59.359169 17719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:59.360541 17719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:59.374549 17724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:59.374500 17725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:59.374466 17728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:59.375314 17719 server_base.cc:1034] running on GCE node
I20260806 22:45:59.375949 17719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:59.376888 17719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:59.378105 17719 hybrid_clock.cc:648] HybridClock initialized: now 1786056359378064 us; error 36 us; skew 500 ppm
I20260806 22:45:59.378436 17719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:59 dist-test-slave-zr1t krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:59.381764 17719 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:45:59.383271 17719 webserver.cc:460] Webserver started at http://127.23.100.130:41633/ using document root <none> and password file <none>
I20260806 22:45:59.383980 17719 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:59.384135 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:59.384459 17719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:45:59.387389 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "421f5795b5d24c408cfb52b111c64c2a"
format_stamp: "Formatted at 2026-08-06 22:45:59 on dist-test-slave-zr1t"
server_key: "ba1a5b506a1c04720f1b537dbc8d7507"
server_key_iv: "3c463b3b799726d868753059b3f99dff"
server_key_version: "encryptionkey@0"
I20260806 22:45:59.388159 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "421f5795b5d24c408cfb52b111c64c2a"
format_stamp: "Formatted at 2026-08-06 22:45:59 on dist-test-slave-zr1t"
server_key: "ba1a5b506a1c04720f1b537dbc8d7507"
server_key_iv: "3c463b3b799726d868753059b3f99dff"
server_key_version: "encryptionkey@0"
I20260806 22:45:59.393402 17719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 22:45:59.396924 17740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:59.398303 17719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:45:59.398483 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "421f5795b5d24c408cfb52b111c64c2a"
format_stamp: "Formatted at 2026-08-06 22:45:59 on dist-test-slave-zr1t"
server_key: "ba1a5b506a1c04720f1b537dbc8d7507"
server_key_iv: "3c463b3b799726d868753059b3f99dff"
server_key_version: "encryptionkey@0"
I20260806 22:45:59.398689 17719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:59.419818 17719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:59.423861 17719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:59.424172 17719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:59.425146 17719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:45:59.426614 17719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:45:59.426723 17719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:59.426882 17719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:45:59.426956 17719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:59.462716 17719 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:42113
I20260806 22:45:59.462764 17916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:42113 every 8 connection(s)
I20260806 22:45:59.464038 17719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:45:59.464354 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 17719
I20260806 22:45:59.464576 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:45:59.465194 23954 external_mini_cluster.cc:1428] Setting key 9030717a40362e582531795796a75f2d
Aug 06 22:45:59 dist-test-slave-zr1t krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:45:59.485639 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:59.466990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:60761 (local address 127.23.100.190:46345)
0806 22:45:59.469902 (+ 2912us) server_negotiation.cc:206] Beginning negotiation
0806 22:45:59.469908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:45:59.469929 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:45:59.471103 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:45:59.471108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:45:59.471196 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:45:59.471315 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:45:59.472950 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:59.473850 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:59.475500 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:45:59.475951 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:45:59.476091 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:59.479734 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:45:59.479740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:45:59.479744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:45:59.479776 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:45:59.482521 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:59.483275 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:59.483283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:59.483292 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:59.483430 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:45:59.483860 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:45:59.483866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:45:59.483869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:45:59.484148 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:45:59.484307 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 22:45:59.484987 (+ 680us) server_negotiation.cc:299] Negotiation successful
0806 22:45:59.485214 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2779,"thread_start_us":2670,"threads_started":1}
I20260806 22:45:59.486624 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:45:59.467724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:46345 (local address 127.23.100.130:60761)
0806 22:45:59.468607 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0806 22:45:59.468685 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 22:45:59.470804 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:45:59.471580 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:45:59.471588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:45:59.472152 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:45:59.472773 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:59.472797 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:59.474014 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:59.474024 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:59.475342 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:45:59.475356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:45:59.476185 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:45:59.476194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:45:59.476443 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:45:59.477149 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:45:59.477178 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:45:59.479499 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:45:59.482677 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:45:59.482686 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:45:59.482701 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:45:59.483133 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:45:59.483546 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:45:59.483551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:45:59.483554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:45:59.483744 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:45:59.484280 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:45:59.484299 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:45:59.484630 (+ 331us) client_negotiation.cc:769] Sending connection context
0806 22:45:59.484913 (+ 283us) client_negotiation.cc:240] Negotiation successful
0806 22:45:59.485448 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":5719,"threads_started":1}
I20260806 22:45:59.488294 17918 heartbeater.cc:347] Connected to a master server at 127.23.100.190:46345
I20260806 22:45:59.488646 17918 heartbeater.cc:464] Registering TS with master...
I20260806 22:45:59.489542 17918 heartbeater.cc:511] Master 127.23.100.190:46345 requested a full tablet report, sending...
I20260806 22:45:59.491307 17427 ts_manager.cc:194] Registered new tserver with Master: 421f5795b5d24c408cfb52b111c64c2a (127.23.100.130:42113)
I20260806 22:45:59.492595 17427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:60761
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:45:59.540354 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:46345
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:45:59.892153 17927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:45:59.892431 17927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:45:59.892525 17927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:45:59.899670 17927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:45:59.899778 17927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:45:59.899890 17927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:45:59.912495 17927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:46345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:45:59.913743 17927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:45:59.915011 17927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:45:59.928624 17934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:59.928663 17935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:45:59.929441 17937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:45:59.929854 17927 server_base.cc:1034] running on GCE node
I20260806 22:45:59.930627 17927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:45:59.931659 17927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:45:59.932941 17927 hybrid_clock.cc:648] HybridClock initialized: now 1786056359932890 us; error 58 us; skew 500 ppm
I20260806 22:45:59.933310 17927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:45:59 dist-test-slave-zr1t krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:45:59.937181 17927 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:45:59.938980 17927 webserver.cc:460] Webserver started at http://127.23.100.131:38531/ using document root <none> and password file <none>
I20260806 22:45:59.939690 17927 fs_manager.cc:359] Metadata directory not provided
I20260806 22:45:59.939807 17927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:45:59.940151 17927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:45:59.940416 17709 heartbeater.cc:503] Master 127.23.100.190:46345 was elected leader, sending a full tablet report...
I20260806 22:45:59.944057 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "d29b1f9c60a440b886d3f9f2ab582af6"
format_stamp: "Formatted at 2026-08-06 22:45:59 on dist-test-slave-zr1t"
server_key: "503f786e95c515dc44d26698db0b6304"
server_key_iv: "bfe4667944b2733e6b96a34beb589df9"
server_key_version: "encryptionkey@0"
I20260806 22:45:59.944928 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "d29b1f9c60a440b886d3f9f2ab582af6"
format_stamp: "Formatted at 2026-08-06 22:45:59 on dist-test-slave-zr1t"
server_key: "503f786e95c515dc44d26698db0b6304"
server_key_iv: "bfe4667944b2733e6b96a34beb589df9"
server_key_version: "encryptionkey@0"
I20260806 22:45:59.950875 17927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 22:45:59.954816 17949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:59.956269 17927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:45:59.956434 17927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "d29b1f9c60a440b886d3f9f2ab582af6"
format_stamp: "Formatted at 2026-08-06 22:45:59 on dist-test-slave-zr1t"
server_key: "503f786e95c515dc44d26698db0b6304"
server_key_iv: "bfe4667944b2733e6b96a34beb589df9"
server_key_version: "encryptionkey@0"
I20260806 22:45:59.956629 17927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:45:59.978461 17927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:45:59.983457 17927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:45:59.983736 17927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:45:59.984846 17927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:45:59.986603 17927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:45:59.986693 17927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:45:59.986797 17927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:45:59.986847 17927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:00.023108 17927 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:44927
I20260806 22:46:00.023164 18135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:44927 every 8 connection(s)
I20260806 22:46:00.024552 17927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:46:00.034657 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 17927
I20260806 22:46:00.034881 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:46:00.035380 23954 external_mini_cluster.cc:1428] Setting key 7a155244bfef3ff66ef84cb2f121492e
Aug 06 22:46:00 dist-test-slave-zr1t krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056359, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:46:00.047080 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.027406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:53621 (local address 127.23.100.190:46345)
0806 22:46:00.027968 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:00.027973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:00.028981 (+ 1008us) server_negotiation.cc:407] Connection header received
0806 22:46:00.031134 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:00.031141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:00.031248 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:00.031388 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:00.033094 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.034089 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.036071 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.036634 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.036791 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.040950 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:00.040957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:00.040960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:00.040999 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:00.044064 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:00.044843 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:00.044847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:00.044849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:00.044991 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:00.045391 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:00.045394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:00.045396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:00.045742 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:00.045867 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:00.046602 (+ 735us) server_negotiation.cc:299] Negotiation successful
0806 22:46:00.046785 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":264,"threads_started":1}
I20260806 22:46:00.048221 18143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.028004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:46345 (local address 127.23.100.131:53621)
0806 22:46:00.028787 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:00.028841 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:00.030821 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:00.031667 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:00.031676 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:00.032252 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:00.032891 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.032910 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.034260 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.034271 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.035897 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.035920 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.036983 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.036992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.037221 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.038138 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:46:00.038167 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:00.040720 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:00.044204 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:00.044212 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:00.044228 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:00.044732 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:00.045078 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:00.045082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:00.045085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:00.045282 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:00.045878 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:00.045885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:00.046239 (+ 354us) client_negotiation.cc:769] Sending connection context
0806 22:46:00.046503 (+ 264us) client_negotiation.cc:240] Negotiation successful
0806 22:46:00.046960 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":318,"threads_started":1}
I20260806 22:46:00.049993 18140 heartbeater.cc:347] Connected to a master server at 127.23.100.190:46345
I20260806 22:46:00.050343 18140 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:00.051393 18140 heartbeater.cc:511] Master 127.23.100.190:46345 requested a full tablet report, sending...
I20260806 22:46:00.053097 17427 ts_manager.cc:194] Registered new tserver with Master: d29b1f9c60a440b886d3f9f2ab582af6 (127.23.100.131:44927)
I20260806 22:46:00.054005 17427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:53621
I20260806 22:46:00.064209 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:46:00.083732 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.069627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.23.100.190:46345)
0806 22:46:00.069880 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:00.069891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:00.070303 (+ 412us) server_negotiation.cc:407] Connection header received
0806 22:46:00.070513 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:00.070521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:00.070607 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:00.070710 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:00.072521 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.073309 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.074813 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.075239 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.075342 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.078088 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:00.078094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:00.078097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:00.078133 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:00.080869 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:00.081726 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:00.081730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:00.081732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:00.081860 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:00.082282 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:00.082285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:00.082288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:00.082634 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:00.082830 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:00.083183 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 22:46:00.083372 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 22:46:00.088295 17427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:46:00.095232 17427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:46:00.121111 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.108714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:42113 (local address 127.0.0.1:37458)
0806 22:46:00.109921 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:00.109990 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:00.110233 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:00.111759 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:00.111770 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:00.111832 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:00.112455 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.112473 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.114960 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.114967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.117768 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.117784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.120031 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.120039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.120249 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.120314 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 22:46:00.120429 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 22:46:00.120652 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":600,"threads_started":1}
I20260806 22:46:00.123564 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.108997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.23.100.130:42113)
0806 22:46:00.110987 (+ 1990us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:00.110997 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:00.111031 (+ 34us) server_negotiation.cc:407] Connection header received
0806 22:46:00.111150 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:00.111162 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:00.111460 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:00.112677 (+ 1217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:00.112970 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.114748 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.118037 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.119763 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.122660 (+ 2897us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.122900 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:00.122982 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 22:46:00.123181 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1779,"thread_start_us":1631,"threads_started":1}
I20260806 22:46:00.123752 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.109011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:44927 (local address 127.0.0.1:37864)
0806 22:46:00.112454 (+ 3443us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:00.112474 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:00.112619 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:00.114267 (+ 1648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:00.114274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:00.114302 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:00.114776 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.114792 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.117080 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.117086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.119550 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.119567 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.122866 (+ 3299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.122874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.123068 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.123107 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 22:46:00.123192 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 22:46:00.123345 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3243,"thread_start_us":2774,"threads_started":1}
I20260806 22:46:00.124817 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.109570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.23.100.131:44927)
0806 22:46:00.111602 (+ 2032us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:00.111613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:00.112744 (+ 1131us) server_negotiation.cc:407] Connection header received
0806 22:46:00.112859 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:00.112867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:00.113768 (+ 901us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:00.113968 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:00.115034 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.116889 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.119771 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.121417 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.121664 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.124172 (+ 2508us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:00.124258 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 22:46:00.124448 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1828,"thread_start_us":1619,"threads_started":1}
I20260806 22:46:00.128527 17817 tablet_service.cc:1467] Processing CreateTablet for tablet a9a9baa2786e420f843246eac5866b43 (DEFAULT_TABLE table=test-table [id=24636bb6709f4a29ba839ca35f8532c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:00.128700 18033 tablet_service.cc:1467] Processing CreateTablet for tablet a9a9baa2786e420f843246eac5866b43 (DEFAULT_TABLE table=test-table [id=24636bb6709f4a29ba839ca35f8532c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:00.130008 17817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a9baa2786e420f843246eac5866b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:00.130308 18033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a9baa2786e420f843246eac5866b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:00.133919 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.112277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:38073 (local address 127.0.0.1:50440)
0806 22:46:00.121412 (+ 9135us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:00.121436 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:00.121577 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:00.125996 (+ 4419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:00.126005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:00.126041 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:00.126240 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.126252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.128399 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.128405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.130522 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.130541 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.132659 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.132666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.132886 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.132928 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 22:46:00.133337 (+ 409us) client_negotiation.cc:240] Negotiation successful
0806 22:46:00.133576 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9002,"thread_start_us":11954,"threads_started":1}
I20260806 22:46:00.134773 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.112393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.23.100.129:38073)
0806 22:46:00.125182 (+ 12789us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:00.125191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:00.125225 (+ 34us) server_negotiation.cc:407] Connection header received
0806 22:46:00.125345 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:00.125356 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:00.125684 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:00.125984 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:00.126443 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.128211 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.130749 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.132430 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.132691 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.132897 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:00.134186 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0806 22:46:00.134406 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12591,"thread_start_us":12432,"threads_started":1}
I20260806 22:46:00.139410 17612 tablet_service.cc:1467] Processing CreateTablet for tablet a9a9baa2786e420f843246eac5866b43 (DEFAULT_TABLE table=test-table [id=24636bb6709f4a29ba839ca35f8532c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:00.141428 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a9baa2786e420f843246eac5866b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:00.144629 18169 tablet_bootstrap.cc:492] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: Bootstrap starting.
I20260806 22:46:00.147038 18173 tablet_bootstrap.cc:492] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: Bootstrap starting.
I20260806 22:46:00.148973 18169 tablet_bootstrap.cc:654] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:00.150589 18169 log.cc:824] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:00.151067 18173 tablet_bootstrap.cc:654] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:00.152895 18173 log.cc:824] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:00.156247 18173 tablet_bootstrap.cc:492] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: No bootstrap required, opened a new log
I20260806 22:46:00.156639 18173 ts_tablet_manager.cc:1397] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260806 22:46:00.156890 18169 tablet_bootstrap.cc:492] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: No bootstrap required, opened a new log
I20260806 22:46:00.157194 18169 ts_tablet_manager.cc:1397] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260806 22:46:00.157491 18177 tablet_bootstrap.cc:492] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: Bootstrap starting.
I20260806 22:46:00.162358 18177 tablet_bootstrap.cc:654] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:00.163947 18177 log.cc:824] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:00.167559 18169 raft_consensus.cc:348] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421f5795b5d24c408cfb52b111c64c2a" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 42113 } } peers { permanent_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 44927 } } peers { permanent_uuid: "82db840e9ef349c99fb633ff5f188c32" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 38073 } }
I20260806 22:46:00.168004 18169 raft_consensus.cc:374] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:00.168174 18177 tablet_bootstrap.cc:492] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: No bootstrap required, opened a new log
I20260806 22:46:00.168118 18169 raft_consensus.cc:729] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29b1f9c60a440b886d3f9f2ab582af6, State: Initialized, Role: FOLLOWER
I20260806 22:46:00.168531 18177 ts_tablet_manager.cc:1397] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260806 22:46:00.168929 18169 consensus_queue.cc:260] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421f5795b5d24c408cfb52b111c64c2a" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 42113 } } peers { permanent_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 44927 } } peers { permanent_uuid: "82db840e9ef349c99fb633ff5f188c32" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 38073 } }
I20260806 22:46:00.168774 18173 raft_consensus.cc:348] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421f5795b5d24c408cfb52b111c64c2a" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 42113 } } peers { permanent_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 44927 } } peers { permanent_uuid: "82db840e9ef349c99fb633ff5f188c32" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 38073 } }
I20260806 22:46:00.169265 18173 raft_consensus.cc:374] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:00.169412 18173 raft_consensus.cc:729] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421f5795b5d24c408cfb52b111c64c2a, State: Initialized, Role: FOLLOWER
I20260806 22:46:00.170400 18140 heartbeater.cc:503] Master 127.23.100.190:46345 was elected leader, sending a full tablet report...
I20260806 22:46:00.170197 18173 consensus_queue.cc:260] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421f5795b5d24c408cfb52b111c64c2a" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 42113 } } peers { permanent_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 44927 } } peers { permanent_uuid: "82db840e9ef349c99fb633ff5f188c32" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 38073 } }
I20260806 22:46:00.171164 18169 ts_tablet_manager.cc:1428] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260806 22:46:00.178397 17918 heartbeater.cc:503] Master 127.23.100.190:46345 was elected leader, sending a full tablet report...
I20260806 22:46:00.179320 18173 ts_tablet_manager.cc:1428] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: Time spent starting tablet: real 0.023s user 0.018s sys 0.005s
I20260806 22:46:00.182430 18177 raft_consensus.cc:348] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421f5795b5d24c408cfb52b111c64c2a" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 42113 } } peers { permanent_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 44927 } } peers { permanent_uuid: "82db840e9ef349c99fb633ff5f188c32" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 38073 } }
I20260806 22:46:00.182825 18177 raft_consensus.cc:374] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:00.182974 18177 raft_consensus.cc:729] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82db840e9ef349c99fb633ff5f188c32, State: Initialized, Role: FOLLOWER
I20260806 22:46:00.183713 18177 consensus_queue.cc:260] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421f5795b5d24c408cfb52b111c64c2a" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 42113 } } peers { permanent_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 44927 } } peers { permanent_uuid: "82db840e9ef349c99fb633ff5f188c32" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 38073 } }
I20260806 22:46:00.185670 18177 ts_tablet_manager.cc:1428] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: Time spent starting tablet: real 0.017s user 0.019s sys 0.000s
W20260806 22:46:00.216351 17920 tablet.cc:2369] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:46:00.276408 18141 tablet.cc:2369] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:46:00.387183 18184 raft_consensus.cc:482] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:46:00.387486 18184 raft_consensus.cc:504] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421f5795b5d24c408cfb52b111c64c2a" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 42113 } } peers { permanent_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 44927 } } peers { permanent_uuid: "82db840e9ef349c99fb633ff5f188c32" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 38073 } }
I20260806 22:46:00.389276 18184 leader_election.cc:288] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d29b1f9c60a440b886d3f9f2ab582af6 (127.23.100.131:44927), 82db840e9ef349c99fb633ff5f188c32 (127.23.100.129:38073)
I20260806 22:46:00.396538 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.389875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:50951 (local address 127.23.100.129:38073)
0806 22:46:00.390142 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:00.390150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:00.391050 (+ 900us) server_negotiation.cc:407] Connection header received
0806 22:46:00.391255 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:00.391259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:00.391317 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:00.391418 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:00.392083 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.392765 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.394749 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.395547 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.395733 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.395818 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:00.396151 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 22:46:00.396280 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 22:46:00.396625 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.389874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:50031 (local address 127.23.100.131:44927)
0806 22:46:00.390130 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:00.390139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:00.390943 (+ 804us) server_negotiation.cc:407] Connection header received
0806 22:46:00.391177 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:00.391183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:00.391246 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:00.391346 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:00.391996 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.392759 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.394773 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.395559 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.395758 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.395845 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:00.396242 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 22:46:00.396367 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 22:46:00.396734 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.389758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:38073 (local address 127.23.100.130:50951)
0806 22:46:00.390922 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:00.390938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:00.391103 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:00.391433 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:00.391438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:00.391476 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:00.391909 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.391919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.392902 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.392908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.394578 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.394595 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.395807 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.395814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.395990 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.396015 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 22:46:00.396124 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 22:46:00.396254 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":25,"thread_start_us":691,"threads_started":1}
I20260806 22:46:00.396785 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.389557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:44927 (local address 127.23.100.130:50031)
0806 22:46:00.390811 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:00.390829 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:00.391025 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:00.391365 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:00.391378 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:00.391405 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:00.391840 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.391851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.392894 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.392903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.394611 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:00.394624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.395848 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.395857 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:00.396069 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.396111 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 22:46:00.396213 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 22:46:00.396338 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"mutex_wait_us":78,"thread_start_us":963,"threads_started":1}
I20260806 22:46:00.397904 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a9baa2786e420f843246eac5866b43" candidate_uuid: "421f5795b5d24c408cfb52b111c64c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" is_pre_election: true
I20260806 22:46:00.397910 17635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a9baa2786e420f843246eac5866b43" candidate_uuid: "421f5795b5d24c408cfb52b111c64c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82db840e9ef349c99fb633ff5f188c32" is_pre_election: true
I20260806 22:46:00.398524 18058 raft_consensus.cc:2454] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 421f5795b5d24c408cfb52b111c64c2a in term 0.
I20260806 22:46:00.398524 17635 raft_consensus.cc:2454] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 421f5795b5d24c408cfb52b111c64c2a in term 0.
I20260806 22:46:00.399292 17745 leader_election.cc:302] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 421f5795b5d24c408cfb52b111c64c2a, 82db840e9ef349c99fb633ff5f188c32; no voters:
I20260806 22:46:00.399744 18184 raft_consensus.cc:2781] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:46:00.399914 18184 raft_consensus.cc:482] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:46:00.400033 18184 raft_consensus.cc:3037] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:00.402390 18184 raft_consensus.cc:504] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421f5795b5d24c408cfb52b111c64c2a" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 42113 } } peers { permanent_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 44927 } } peers { permanent_uuid: "82db840e9ef349c99fb633ff5f188c32" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 38073 } }
I20260806 22:46:00.403127 18184 leader_election.cc:288] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [CANDIDATE]: Term 1 election: Requested vote from peers d29b1f9c60a440b886d3f9f2ab582af6 (127.23.100.131:44927), 82db840e9ef349c99fb633ff5f188c32 (127.23.100.129:38073)
I20260806 22:46:00.403674 18058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a9baa2786e420f843246eac5866b43" candidate_uuid: "421f5795b5d24c408cfb52b111c64c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29b1f9c60a440b886d3f9f2ab582af6"
I20260806 22:46:00.403669 17635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a9baa2786e420f843246eac5866b43" candidate_uuid: "421f5795b5d24c408cfb52b111c64c2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82db840e9ef349c99fb633ff5f188c32"
I20260806 22:46:00.403877 18058 raft_consensus.cc:3037] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:00.403894 17635 raft_consensus.cc:3037] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:00.406227 17635 raft_consensus.cc:2454] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 421f5795b5d24c408cfb52b111c64c2a in term 1.
I20260806 22:46:00.406255 18058 raft_consensus.cc:2454] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 421f5795b5d24c408cfb52b111c64c2a in term 1.
I20260806 22:46:00.406698 17741 leader_election.cc:302] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 421f5795b5d24c408cfb52b111c64c2a, d29b1f9c60a440b886d3f9f2ab582af6; no voters:
I20260806 22:46:00.407045 18184 raft_consensus.cc:2781] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:00.407593 18184 raft_consensus.cc:686] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 1 LEADER]: Becoming Leader. State: Replica: 421f5795b5d24c408cfb52b111c64c2a, State: Running, Role: LEADER
I20260806 22:46:00.408140 18184 consensus_queue.cc:237] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421f5795b5d24c408cfb52b111c64c2a" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 42113 } } peers { permanent_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 44927 } } peers { permanent_uuid: "82db840e9ef349c99fb633ff5f188c32" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 38073 } }
W20260806 22:46:00.410259 17712 tablet.cc:2369] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:46:00.413057 17427 catalog_manager.cc:5495] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 421f5795b5d24c408cfb52b111c64c2a (127.23.100.130). New cstate: current_term: 1 leader_uuid: "421f5795b5d24c408cfb52b111c64c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421f5795b5d24c408cfb52b111c64c2a" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 42113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 44927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82db840e9ef349c99fb633ff5f188c32" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 38073 } health_report { overall_health: UNKNOWN } } }
I20260806 22:46:00.491115 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.485373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.23.100.130:42113)
0806 22:46:00.485637 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:00.485644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:00.485762 (+ 118us) server_negotiation.cc:407] Connection header received
0806 22:46:00.486030 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:00.486034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:00.486122 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:00.486277 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:46:00.486681 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.487394 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.488834 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.489246 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.489394 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.490012 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:46:00.490499 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:46:00.490658 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:00.490742 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 22:46:00.490824 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 22:46:00.499449 18058 raft_consensus.cc:1261] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6 [term 1 FOLLOWER]: Refusing update from remote peer 421f5795b5d24c408cfb52b111c64c2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:00.499464 17635 raft_consensus.cc:1261] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32 [term 1 FOLLOWER]: Refusing update from remote peer 421f5795b5d24c408cfb52b111c64c2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:00.500523 18190 consensus_queue.cc:1035] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29b1f9c60a440b886d3f9f2ab582af6" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 44927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:46:00.500892 18184 consensus_queue.cc:1035] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "82db840e9ef349c99fb633ff5f188c32" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:46:00.513237 18195 mvcc.cc:205] Tried to move back new op lower bound from 7315686852595052544 to 7315686852235747328. Current Snapshot: MvccSnapshot[applied={T|T < 7315686852595052544}]
I20260806 22:46:00.516469 18196 mvcc.cc:205] Tried to move back new op lower bound from 7315686852595052544 to 7315686852235747328. Current Snapshot: MvccSnapshot[applied={T|T < 7315686852595052544}]
I20260806 22:46:00.527648 17427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:00.527938 17427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:00.531477 17427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 098664a50b2b40fd9c3ed77924ec2259: Sending DeleteTablet for 3 replicas of tablet a9a9baa2786e420f843246eac5866b43
I20260806 22:46:00.532634 17817 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a9baa2786e420f843246eac5866b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:00 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:37458
I20260806 22:46:00.532721 18033 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a9baa2786e420f843246eac5866b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:00 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:37864
I20260806 22:46:00.532950 17612 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a9baa2786e420f843246eac5866b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:00 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:50440
I20260806 22:46:00.535583 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 17523
I20260806 22:46:00.566220 18214 tablet_replica.cc:330] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: stopping tablet replica
I20260806 22:46:00.567642 18214 raft_consensus.cc:2229] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:00.569916 18214 raft_consensus.cc:2258] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:00.574152 18210 tablet_replica.cc:330] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: stopping tablet replica
I20260806 22:46:00.575512 18211 tablet_replica.cc:330] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: stopping tablet replica
I20260806 22:46:00.575966 18211 raft_consensus.cc:2229] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:00.576162 18210 raft_consensus.cc:2229] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:46:00.576519 18211 raft_consensus.cc:2258] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:00.577001 18210 raft_consensus.cc:2258] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:00.578222 18214 ts_tablet_manager.cc:1905] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:00.579486 18210 ts_tablet_manager.cc:1905] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 22:46:00 dist-test-slave-zr1t krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:46:00.584292 18211 ts_tablet_manager.cc:1905] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:00.586122 18214 ts_tablet_manager.cc:1918] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:00.586297 18214 log.cc:1196] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/a9a9baa2786e420f843246eac5866b43
I20260806 22:46:00.586894 18214 ts_tablet_manager.cc:1939] T a9a9baa2786e420f843246eac5866b43 P 82db840e9ef349c99fb633ff5f188c32: Deleting consensus metadata
I20260806 22:46:00.588006 18210 ts_tablet_manager.cc:1918] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:00.588275 18210 log.cc:1196] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/a9a9baa2786e420f843246eac5866b43
I20260806 22:46:00.588972 18210 ts_tablet_manager.cc:1939] T a9a9baa2786e420f843246eac5866b43 P 421f5795b5d24c408cfb52b111c64c2a: Deleting consensus metadata
I20260806 22:46:00.588928 17402 catalog_manager.cc:4841] TS 82db840e9ef349c99fb633ff5f188c32 (127.23.100.129:38073): tablet a9a9baa2786e420f843246eac5866b43 (table test-table [id=24636bb6709f4a29ba839ca35f8532c6]) successfully deleted
I20260806 22:46:00.590867 17401 catalog_manager.cc:4841] TS 421f5795b5d24c408cfb52b111c64c2a (127.23.100.130:42113): tablet a9a9baa2786e420f843246eac5866b43 (table test-table [id=24636bb6709f4a29ba839ca35f8532c6]) successfully deleted
I20260806 22:46:00.591602 18211 ts_tablet_manager.cc:1918] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:00.591806 18211 log.cc:1196] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/a9a9baa2786e420f843246eac5866b43
I20260806 22:46:00.592262 18211 ts_tablet_manager.cc:1939] T a9a9baa2786e420f843246eac5866b43 P d29b1f9c60a440b886d3f9f2ab582af6: Deleting consensus metadata
I20260806 22:46:00.593712 17399 catalog_manager.cc:4841] TS d29b1f9c60a440b886d3f9f2ab582af6 (127.23.100.131:44927): tablet a9a9baa2786e420f843246eac5866b43 (table test-table [id=24636bb6709f4a29ba839ca35f8532c6]) successfully deleted
I20260806 22:46:00.596851 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.537933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58218 (local address 127.23.100.129:38073)
0806 22:46:00.538165 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:00.538174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:00.563600 (+ 25426us) server_negotiation.cc:407] Connection header received
0806 22:46:00.564264 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:00.564274 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:00.564400 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:00.564499 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:00.570498 (+ 5999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.572064 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.575364 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.576024 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.576160 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.589583 (+ 13423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:00.589599 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:00.589621 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:00.589675 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:00.593830 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:00.594620 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:00.594629 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:00.594633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:00.594800 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:00.595216 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:00.595221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:00.595225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:00.595677 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:00.595992 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:00.596225 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 22:46:00.596471 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 22:46:00.597865 17698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:00.696059 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 17523
I20260806 22:46:00.727463 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 17719
Aug 06 22:46:00 dist-test-slave-zr1t krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:46:00.750597 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.728324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.23.100.130:42113)
0806 22:46:00.728527 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:00.728532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:00.728727 (+ 195us) server_negotiation.cc:407] Connection header received
0806 22:46:00.728882 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:00.728888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:00.728976 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:00.729100 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:00.730435 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.731685 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.733068 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.733967 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.734094 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.743319 (+ 9225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:00.743341 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:00.743363 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:00.743418 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:00.747573 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:00.748316 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:00.748331 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:00.748335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:00.748513 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:00.748883 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:00.748889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:00.748892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:00.749336 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:00.749721 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:00.749929 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 22:46:00.750168 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 22:46:00.751435 17912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:00.871824 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 17719
I20260806 22:46:00.897462 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 17927
Aug 06 22:46:00 dist-test-slave-zr1t krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:46:00.913780 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:00.898309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.23.100.131:44927)
0806 22:46:00.898821 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:00.898826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:00.898843 (+ 17us) server_negotiation.cc:407] Connection header received
0806 22:46:00.898901 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:00.898905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:00.898968 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:00.899063 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:00.900349 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.901097 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.902614 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:00.903008 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:00.903104 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:00.907786 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:00.907804 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:00.907820 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:00.907865 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:00.910787 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:00.911548 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:00.911561 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:00.911564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:00.911707 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:00.912114 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:00.912120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:00.912125 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:00.912511 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:00.912825 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:00.913168 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 22:46:00.913401 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":295,"threads_started":1}
I20260806 22:46:00.914463 18127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:01.003830 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 17927
I20260806 22:46:01.030483 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 17374
I20260806 22:46:01.031811 17475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:01.146175 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 17374
2026-08-06T22:46:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5159 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T22:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:01Z Disabled control of system clock
I20260806 22:46:01.255429 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:37783
--webserver_interface=127.23.100.190
--webserver_port=0
--builtin_ntp_servers=127.23.100.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:37783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 22:46:01.607506 18232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:01.607798 18232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:01.607981 18232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:01.615351 18232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:01.615461 18232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:01.615536 18232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:01.615629 18232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:01.615732 18232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:01.625900 18232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:37783
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:37783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:01.627202 18232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:01.628981 18232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:01.641135 18240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:01.641240 18241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:01.641292 18245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:01.642324 18232 server_base.cc:1034] running on GCE node
I20260806 22:46:01.643141 18232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:01.644557 18232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:01.645798 18232 hybrid_clock.cc:648] HybridClock initialized: now 1786056361645743 us; error 41 us; skew 500 ppm
I20260806 22:46:01.646175 18232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:01.648449 18232 webserver.cc:460] Webserver started at http://127.23.100.190:35991/ using document root <none> and password file <none>
I20260806 22:46:01.649171 18232 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:01.649308 18232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:01.649703 18232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:01.652725 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "1ce889ee9d3544a78de984398438f383"
format_stamp: "Formatted at 2026-08-06 22:46:01 on dist-test-slave-zr1t"
server_key: "db3bcac09fff9b4b9aff689248a1ddf3"
server_key_iv: "36059392a95a7670a70ffd852790e538"
server_key_version: "encryptionkey@0"
I20260806 22:46:01.653643 18232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "1ce889ee9d3544a78de984398438f383"
format_stamp: "Formatted at 2026-08-06 22:46:01 on dist-test-slave-zr1t"
server_key: "db3bcac09fff9b4b9aff689248a1ddf3"
server_key_iv: "36059392a95a7670a70ffd852790e538"
server_key_version: "encryptionkey@0"
I20260806 22:46:01.658864 18232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:46:01.662472 18254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:01.663832 18232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:46:01.664040 18232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "1ce889ee9d3544a78de984398438f383"
format_stamp: "Formatted at 2026-08-06 22:46:01 on dist-test-slave-zr1t"
server_key: "db3bcac09fff9b4b9aff689248a1ddf3"
server_key_iv: "36059392a95a7670a70ffd852790e538"
server_key_version: "encryptionkey@0"
I20260806 22:46:01.664268 18232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:01.683527 18232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:01.684383 18232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:01.684687 18232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:01.703361 18232 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:37783
I20260806 22:46:01.703401 18331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:37783 every 8 connection(s)
I20260806 22:46:01.704733 18232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:01.707653 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 18232
I20260806 22:46:01.707844 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:01.708339 23954 external_mini_cluster.cc:1428] Setting key f111e0eab5d5b161b0d542b8628bf7d9
I20260806 22:46:01.708686 18333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:01.724658 18333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383: Bootstrap starting.
I20260806 22:46:01.724890 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:01.710398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.23.100.190:37783)
0806 22:46:01.711503 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:01.711517 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:01.711577 (+ 60us) server_negotiation.cc:407] Connection header received
0806 22:46:01.713803 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:01.713834 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:01.714480 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:01.714952 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:01.715547 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:01.717694 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:01.719201 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:01.720295 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:01.720515 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:01.721881 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:01.721892 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:01.721912 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:01.722708 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:01.722810 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:01.722837 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:01.722958 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:01.723229 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 22:46:01.723587 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":337,"threads_started":1}
I20260806 22:46:01.728569 18333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:01.729626 18333 log.cc:824] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:01.732897 18333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383: No bootstrap required, opened a new log
I20260806 22:46:01.741452 18333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce889ee9d3544a78de984398438f383" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } }
I20260806 22:46:01.742062 18333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:01.742193 18333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce889ee9d3544a78de984398438f383, State: Initialized, Role: FOLLOWER
I20260806 22:46:01.742961 18333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce889ee9d3544a78de984398438f383" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } }
I20260806 22:46:01.743253 18333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:01.743422 18333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:01.743602 18333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:01.745960 18333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce889ee9d3544a78de984398438f383" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } }
I20260806 22:46:01.746482 18333 leader_election.cc:302] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ce889ee9d3544a78de984398438f383; no voters:
I20260806 22:46:01.747162 18333 leader_election.cc:288] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:01.747375 18343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:01.748129 18343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [term 1 LEADER]: Becoming Leader. State: Replica: 1ce889ee9d3544a78de984398438f383, State: Running, Role: LEADER
I20260806 22:46:01.748648 18343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce889ee9d3544a78de984398438f383" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } }
I20260806 22:46:01.749322 18333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:01.751310 18344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ce889ee9d3544a78de984398438f383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce889ee9d3544a78de984398438f383" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } } }
I20260806 22:46:01.751730 18344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:01.752507 18345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ce889ee9d3544a78de984398438f383. Latest consensus state: current_term: 1 leader_uuid: "1ce889ee9d3544a78de984398438f383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce889ee9d3544a78de984398438f383" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 37783 } } }
I20260806 22:46:01.752889 18345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:01.755129 18352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:01.760854 18352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:01.771512 18352 catalog_manager.cc:1347] Generated new cluster ID: 571e838be3a3487a8a1269d4e4426c7e
I20260806 22:46:01.771649 18352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:01.783567 18352 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:01.784878 18352 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:01.792882 18352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383: Generated new TSK 0
I20260806 22:46:01.793798 18352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:01.809412 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--builtin_ntp_servers=127.23.100.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 22:46:02.171180 18372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:02.171577 18372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:02.171751 18372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:02.179522 18372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:02.179646 18372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:02.179823 18372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:46:02.189646 18372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:02.191005 18372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:02.192845 18372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:02.206370 18379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:02.206347 18382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:02.206295 18380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:02.207358 18372 server_base.cc:1034] running on GCE node
I20260806 22:46:02.207942 18372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:02.208891 18372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:02.210093 18372 hybrid_clock.cc:648] HybridClock initialized: now 1786056362210048 us; error 33 us; skew 500 ppm
I20260806 22:46:02.210388 18372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:02.212687 18372 webserver.cc:460] Webserver started at http://127.23.100.129:38435/ using document root <none> and password file <none>
I20260806 22:46:02.213349 18372 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:02.213439 18372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:02.213781 18372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:02.216697 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "1815a7718a854d438ac90252a8d971b5"
format_stamp: "Formatted at 2026-08-06 22:46:02 on dist-test-slave-zr1t"
server_key: "00ab2b1d6d5c56b141f8de7a0795caa4"
server_key_iv: "2ddcfa93b1f31b8f13c01272cf38ab96"
server_key_version: "encryptionkey@0"
I20260806 22:46:02.217465 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "1815a7718a854d438ac90252a8d971b5"
format_stamp: "Formatted at 2026-08-06 22:46:02 on dist-test-slave-zr1t"
server_key: "00ab2b1d6d5c56b141f8de7a0795caa4"
server_key_iv: "2ddcfa93b1f31b8f13c01272cf38ab96"
server_key_version: "encryptionkey@0"
I20260806 22:46:02.222566 18372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 22:46:02.226101 18393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:02.227399 18372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:46:02.227566 18372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "1815a7718a854d438ac90252a8d971b5"
format_stamp: "Formatted at 2026-08-06 22:46:02 on dist-test-slave-zr1t"
server_key: "00ab2b1d6d5c56b141f8de7a0795caa4"
server_key_iv: "2ddcfa93b1f31b8f13c01272cf38ab96"
server_key_version: "encryptionkey@0"
I20260806 22:46:02.227790 18372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:02.244323 18372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:02.245208 18372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:02.245486 18372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:02.246512 18372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:02.247978 18372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:02.248072 18372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:02.248148 18372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:02.248203 18372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:02.283367 18372 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:40477
I20260806 22:46:02.283464 18567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:40477 every 8 connection(s)
I20260806 22:46:02.284770 18372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:46:02.285730 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 18372
I20260806 22:46:02.285919 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:46:02.286454 23954 external_mini_cluster.cc:1428] Setting key 2a81013747767c9b6bd2f4502dbfe08e
I20260806 22:46:02.289395 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--builtin_ntp_servers=127.23.100.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 22:46:02.298895 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:02.287597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:59989 (local address 127.23.100.190:37783)
0806 22:46:02.288093 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:02.288100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:02.289601 (+ 1501us) server_negotiation.cc:407] Connection header received
0806 22:46:02.291589 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:02.291598 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:02.291713 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:02.291830 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:02.293146 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:02.294007 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:02.295756 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:02.296224 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:02.296369 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:02.297946 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:02.297952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:02.297961 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:02.297982 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:02.298026 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:02.298040 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:02.298149 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:02.298391 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 22:46:02.298500 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260806 22:46:02.299966 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:02.288299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37783 (local address 127.23.100.129:59989)
0806 22:46:02.289403 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:02.289450 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:02.291298 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:02.292099 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:02.292108 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:02.292194 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:02.292938 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:02.292958 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:02.294184 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:02.294189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:02.295598 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:02.295612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:02.296484 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:02.296493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:02.296734 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:02.297566 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:46:02.297571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:02.297625 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:46:02.297717 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:02.298151 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:02.298166 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:02.298177 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 22:46:02.298380 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 22:46:02.298831 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":373,"threads_started":1}
I20260806 22:46:02.301558 18569 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37783
I20260806 22:46:02.301894 18569 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:02.302829 18569 heartbeater.cc:511] Master 127.23.100.190:37783 requested a full tablet report, sending...
I20260806 22:46:02.305120 18280 ts_manager.cc:194] Registered new tserver with Master: 1815a7718a854d438ac90252a8d971b5 (127.23.100.129:40477)
I20260806 22:46:02.306668 18280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.129:59989
W20260806 22:46:02.639695 18578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:02.640041 18578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:02.640211 18578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:02.647822 18578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:02.647981 18578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:02.648171 18578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:46:02.658104 18578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:02.659456 18578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:02.661341 18578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:02.674958 18585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:02.674938 18586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:02.675114 18591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:02.675813 18578 server_base.cc:1034] running on GCE node
I20260806 22:46:02.676524 18578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:02.677624 18578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:02.678849 18578 hybrid_clock.cc:648] HybridClock initialized: now 1786056362678805 us; error 41 us; skew 500 ppm
I20260806 22:46:02.679203 18578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:02.681608 18578 webserver.cc:460] Webserver started at http://127.23.100.130:44467/ using document root <none> and password file <none>
I20260806 22:46:02.682354 18578 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:02.682476 18578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:02.682818 18578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:02.685866 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "3fe9a8212be143e889dea01a0c94e2d0"
format_stamp: "Formatted at 2026-08-06 22:46:02 on dist-test-slave-zr1t"
server_key: "8eff35b966857c6d79702e61148b4133"
server_key_iv: "5f8498d393075066ffabd7d54bc5184c"
server_key_version: "encryptionkey@0"
I20260806 22:46:02.686684 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "3fe9a8212be143e889dea01a0c94e2d0"
format_stamp: "Formatted at 2026-08-06 22:46:02 on dist-test-slave-zr1t"
server_key: "8eff35b966857c6d79702e61148b4133"
server_key_iv: "5f8498d393075066ffabd7d54bc5184c"
server_key_version: "encryptionkey@0"
I20260806 22:46:02.691916 18578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:46:02.695497 18601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:02.696816 18578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 22:46:02.696986 18578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "3fe9a8212be143e889dea01a0c94e2d0"
format_stamp: "Formatted at 2026-08-06 22:46:02 on dist-test-slave-zr1t"
server_key: "8eff35b966857c6d79702e61148b4133"
server_key_iv: "5f8498d393075066ffabd7d54bc5184c"
server_key_version: "encryptionkey@0"
I20260806 22:46:02.697212 18578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:02.718396 18578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:02.719431 18578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:02.719755 18578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:02.720772 18578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:02.722330 18578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:02.722453 18578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:02.722589 18578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:02.722681 18578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:02.758450 18578 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:44399
I20260806 22:46:02.758513 18779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:44399 every 8 connection(s)
I20260806 22:46:02.759915 18578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:46:02.765578 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 18578
I20260806 22:46:02.765808 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:46:02.766393 23954 external_mini_cluster.cc:1428] Setting key a4d51f934caf5647535a044b3ea16b19
I20260806 22:46:02.770025 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--builtin_ntp_servers=127.23.100.148:41949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 22:46:02.774791 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:02.762713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:58751 (local address 127.23.100.190:37783)
0806 22:46:02.762945 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:02.762949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:02.764355 (+ 1406us) server_negotiation.cc:407] Connection header received
0806 22:46:02.766365 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:02.766375 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:02.766476 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:02.766603 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:02.767920 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:02.769075 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:02.771648 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:02.772125 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:02.772272 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:02.774034 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:02.774039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:02.774042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:02.774061 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:02.774095 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:02.774107 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:02.774188 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:02.774473 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 22:46:02.774561 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 22:46:02.776376 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:02.763229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37783 (local address 127.23.100.130:58751)
0806 22:46:02.764158 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:02.764201 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:02.766072 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:02.766857 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:02.766866 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:02.766959 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:02.767735 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:02.767756 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:02.769235 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:02.769241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:02.771476 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:02.771496 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:02.772374 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:02.772384 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:02.772731 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:02.773618 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:46:02.773624 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:02.773686 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:46:02.773808 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:02.774215 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:02.774229 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:02.774239 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 22:46:02.774435 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 22:46:02.774926 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":293,"threads_started":1}
I20260806 22:46:02.778040 18780 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37783
I20260806 22:46:02.778396 18780 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:02.779218 18780 heartbeater.cc:511] Master 127.23.100.190:37783 requested a full tablet report, sending...
I20260806 22:46:02.780730 18280 ts_manager.cc:194] Registered new tserver with Master: 3fe9a8212be143e889dea01a0c94e2d0 (127.23.100.130:44399)
I20260806 22:46:02.781611 18280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.130:58751
W20260806 22:46:03.124356 18784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:03.124709 18784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:03.124883 18784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:03.132365 18784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:03.132515 18784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:03.132731 18784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:46:03.142364 18784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:41949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:37783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:03.143795 18784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:03.145725 18784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:03.159534 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:03.159726 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:03.159767 18797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:03.160773 18784 server_base.cc:1034] running on GCE node
I20260806 22:46:03.161469 18784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:03.162509 18784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:03.163717 18784 hybrid_clock.cc:648] HybridClock initialized: now 1786056363163647 us; error 61 us; skew 500 ppm
I20260806 22:46:03.164062 18784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:03.166832 18784 webserver.cc:460] Webserver started at http://127.23.100.131:35779/ using document root <none> and password file <none>
I20260806 22:46:03.167697 18784 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:03.167794 18784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:03.168205 18784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:03.171418 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "244f104a414e4a93b54c7e093b64e712"
format_stamp: "Formatted at 2026-08-06 22:46:03 on dist-test-slave-zr1t"
server_key: "7a60c74f879bbe9a2ce2b5838a470f04"
server_key_iv: "a8b52b07aaed4c93a3c40a67cb3ebdec"
server_key_version: "encryptionkey@0"
I20260806 22:46:03.172413 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "244f104a414e4a93b54c7e093b64e712"
format_stamp: "Formatted at 2026-08-06 22:46:03 on dist-test-slave-zr1t"
server_key: "7a60c74f879bbe9a2ce2b5838a470f04"
server_key_iv: "a8b52b07aaed4c93a3c40a67cb3ebdec"
server_key_version: "encryptionkey@0"
I20260806 22:46:03.178248 18784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 22:46:03.182190 18804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:03.183703 18784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 22:46:03.183910 18784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "244f104a414e4a93b54c7e093b64e712"
format_stamp: "Formatted at 2026-08-06 22:46:03 on dist-test-slave-zr1t"
server_key: "7a60c74f879bbe9a2ce2b5838a470f04"
server_key_iv: "a8b52b07aaed4c93a3c40a67cb3ebdec"
server_key_version: "encryptionkey@0"
I20260806 22:46:03.184173 18784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:03.198577 18784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:03.199446 18784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:03.199769 18784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:03.200889 18784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:03.202739 18784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:03.202832 18784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:03.203011 18784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:03.203104 18784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:03.239123 18784 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:35447
I20260806 22:46:03.239291 18976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:35447 every 8 connection(s)
I20260806 22:46:03.240557 18784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:46:03.245551 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 18784
I20260806 22:46:03.245767 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:46:03.246452 23954 external_mini_cluster.cc:1428] Setting key 504aed65adb194b006c89fa9a06d252e
I20260806 22:46:03.254788 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.243321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:35621 (local address 127.23.100.190:37783)
0806 22:46:03.243544 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:03.243550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:03.244948 (+ 1398us) server_negotiation.cc:407] Connection header received
0806 22:46:03.247632 (+ 2684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:03.247638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:03.247738 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:03.247868 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:03.249015 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.250245 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.251956 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.252421 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.252531 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.253966 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:03.253971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:03.253974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:03.253993 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:03.254027 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:03.254039 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:03.254121 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:03.254395 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 22:46:03.254506 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 22:46:03.255798 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.243772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:37783 (local address 127.23.100.131:35621)
0806 22:46:03.244755 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:03.244802 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:03.247337 (+ 2535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:03.248072 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:03.248079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:03.248203 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:03.248840 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.248860 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.250444 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.250448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.251805 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.251818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.252668 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.252674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.252887 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.253654 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:46:03.253656 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:03.253697 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:46:03.253777 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:03.254132 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:03.254140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:03.254146 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 22:46:03.254294 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 22:46:03.254634 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":296,"threads_started":1}
I20260806 22:46:03.257390 18977 heartbeater.cc:347] Connected to a master server at 127.23.100.190:37783
I20260806 22:46:03.257776 18977 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:03.258610 18977 heartbeater.cc:511] Master 127.23.100.190:37783 requested a full tablet report, sending...
I20260806 22:46:03.260237 18280 ts_manager.cc:194] Registered new tserver with Master: 244f104a414e4a93b54c7e093b64e712 (127.23.100.131:35447)
I20260806 22:46:03.261268 18280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.131:35621
I20260806 22:46:03.262372 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:46:03.272724 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.266739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.23.100.190:37783)
0806 22:46:03.266932 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:03.266937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:03.267457 (+ 520us) server_negotiation.cc:407] Connection header received
0806 22:46:03.267582 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:03.267586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:03.267664 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:03.267750 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:03.268249 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.268983 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.270482 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.270892 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.270991 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.272067 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:03.272071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:03.272074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:03.272092 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:03.272120 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:03.272131 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:03.272200 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:03.272343 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 22:46:03.272494 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 22:46:03.276629 18279 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:46:03.283716 18279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:46:03.308208 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.296142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:40477 (local address 127.0.0.1:34432)
0806 22:46:03.297553 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:03.297615 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:03.297820 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:03.299768 (+ 1948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:03.299778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:03.299811 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:03.300234 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.300248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.302539 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.302544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.305189 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.305205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.307206 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.307213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.307427 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.307482 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 22:46:03.307581 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 22:46:03.307782 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":873,"threads_started":1}
I20260806 22:46:03.308723 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.296432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.23.100.129:40477)
0806 22:46:03.299095 (+ 2663us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:03.299105 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:03.299139 (+ 34us) server_negotiation.cc:407] Connection header received
0806 22:46:03.299244 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:03.299254 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:03.299515 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:03.300426 (+ 911us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:03.300671 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.302383 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.305472 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.306982 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.307885 (+ 903us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.308096 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:03.308169 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 22:46:03.308356 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2478,"thread_start_us":2353,"threads_started":1}
I20260806 22:46:03.308894 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.297136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.23.100.131:35447)
0806 22:46:03.298622 (+ 1486us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:03.298635 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:03.299577 (+ 942us) server_negotiation.cc:407] Connection header received
0806 22:46:03.299880 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:03.299888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:03.300098 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:03.300267 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:03.301978 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.303539 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.305589 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.306795 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.306994 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.307238 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:03.308313 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0806 22:46:03.308497 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1257,"thread_start_us":1084,"threads_started":1}
I20260806 22:46:03.308239 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.296441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:35447 (local address 127.0.0.1:32884)
0806 22:46:03.299445 (+ 3004us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:03.299476 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:03.299697 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:03.300608 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:03.300612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:03.300637 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:03.301086 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.301098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.303719 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.303724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.305316 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.305359 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.307401 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.307408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.307591 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.307614 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 22:46:03.307759 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 22:46:03.307909 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2869,"thread_start_us":5019,"threads_started":1}
I20260806 22:46:03.309736 18569 heartbeater.cc:503] Master 127.23.100.190:37783 was elected leader, sending a full tablet report...
I20260806 22:46:03.311638 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.297894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:44399 (local address 127.0.0.1:52578)
0806 22:46:03.301717 (+ 3823us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:03.301735 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:03.301839 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:03.303216 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:03.303221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:03.303250 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:03.303835 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.303849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.305760 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.305766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.307825 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.307838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.310904 (+ 3066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.310911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.311112 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.311143 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 22:46:03.311231 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 22:46:03.311360 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3713,"thread_start_us":1274,"threads_started":1}
I20260806 22:46:03.312525 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.297937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.23.100.130:44399)
0806 22:46:03.302179 (+ 4242us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:03.302186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:03.302210 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:46:03.302299 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:03.302304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:03.302482 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:03.302959 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:03.304041 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.305612 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.308047 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.310376 (+ 2329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.310549 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.310752 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:03.312005 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0806 22:46:03.312183 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4080,"thread_start_us":3963,"threads_started":1}
I20260806 22:46:03.312951 18888 tablet_service.cc:1467] Processing CreateTablet for tablet 196a7d893e6e4c07b317d9c38fd9c6a8 (DEFAULT_TABLE table=test-table [id=5b8867bb845d4d4fa791a85f39e1a647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:03.312949 18469 tablet_service.cc:1467] Processing CreateTablet for tablet 196a7d893e6e4c07b317d9c38fd9c6a8 (DEFAULT_TABLE table=test-table [id=5b8867bb845d4d4fa791a85f39e1a647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:03.314500 18469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196a7d893e6e4c07b317d9c38fd9c6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:03.314863 18888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196a7d893e6e4c07b317d9c38fd9c6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:03.316146 18674 tablet_service.cc:1467] Processing CreateTablet for tablet 196a7d893e6e4c07b317d9c38fd9c6a8 (DEFAULT_TABLE table=test-table [id=5b8867bb845d4d4fa791a85f39e1a647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:03.317591 18674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196a7d893e6e4c07b317d9c38fd9c6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:03.327234 19010 tablet_bootstrap.cc:492] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5: Bootstrap starting.
I20260806 22:46:03.329917 19011 tablet_bootstrap.cc:492] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: Bootstrap starting.
I20260806 22:46:03.331037 19010 tablet_bootstrap.cc:654] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:03.332649 19012 tablet_bootstrap.cc:492] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0: Bootstrap starting.
I20260806 22:46:03.332401 19010 log.cc:824] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:03.333721 19011 tablet_bootstrap.cc:654] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:03.335013 19011 log.cc:824] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:03.336256 19012 tablet_bootstrap.cc:654] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:03.336925 19010 tablet_bootstrap.cc:492] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5: No bootstrap required, opened a new log
I20260806 22:46:03.337214 19010 ts_tablet_manager.cc:1397] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 22:46:03.337507 19012 log.cc:824] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:03.337697 19011 tablet_bootstrap.cc:492] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: No bootstrap required, opened a new log
I20260806 22:46:03.337953 19011 ts_tablet_manager.cc:1397] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260806 22:46:03.340619 19012 tablet_bootstrap.cc:492] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0: No bootstrap required, opened a new log
I20260806 22:46:03.340904 19012 ts_tablet_manager.cc:1397] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 22:46:03.346863 19010 raft_consensus.cc:348] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1815a7718a854d438ac90252a8d971b5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 40477 } } peers { permanent_uuid: "244f104a414e4a93b54c7e093b64e712" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35447 } } peers { permanent_uuid: "3fe9a8212be143e889dea01a0c94e2d0" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44399 } }
I20260806 22:46:03.347270 19010 raft_consensus.cc:374] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:03.347384 19010 raft_consensus.cc:729] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1815a7718a854d438ac90252a8d971b5, State: Initialized, Role: FOLLOWER
I20260806 22:46:03.347549 19011 raft_consensus.cc:348] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1815a7718a854d438ac90252a8d971b5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 40477 } } peers { permanent_uuid: "244f104a414e4a93b54c7e093b64e712" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35447 } } peers { permanent_uuid: "3fe9a8212be143e889dea01a0c94e2d0" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44399 } }
I20260806 22:46:03.347980 19011 raft_consensus.cc:374] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:03.348079 19011 raft_consensus.cc:729] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244f104a414e4a93b54c7e093b64e712, State: Initialized, Role: FOLLOWER
I20260806 22:46:03.348088 19010 consensus_queue.cc:260] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1815a7718a854d438ac90252a8d971b5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 40477 } } peers { permanent_uuid: "244f104a414e4a93b54c7e093b64e712" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35447 } } peers { permanent_uuid: "3fe9a8212be143e889dea01a0c94e2d0" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44399 } }
I20260806 22:46:03.348651 19011 consensus_queue.cc:260] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1815a7718a854d438ac90252a8d971b5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 40477 } } peers { permanent_uuid: "244f104a414e4a93b54c7e093b64e712" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35447 } } peers { permanent_uuid: "3fe9a8212be143e889dea01a0c94e2d0" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44399 } }
I20260806 22:46:03.350044 19010 ts_tablet_manager.cc:1428] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5: Time spent starting tablet: real 0.013s user 0.009s sys 0.006s
I20260806 22:46:03.354072 19012 raft_consensus.cc:348] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1815a7718a854d438ac90252a8d971b5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 40477 } } peers { permanent_uuid: "244f104a414e4a93b54c7e093b64e712" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35447 } } peers { permanent_uuid: "3fe9a8212be143e889dea01a0c94e2d0" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44399 } }
I20260806 22:46:03.354665 19012 raft_consensus.cc:374] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:03.354794 19012 raft_consensus.cc:729] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe9a8212be143e889dea01a0c94e2d0, State: Initialized, Role: FOLLOWER
I20260806 22:46:03.355624 19012 consensus_queue.cc:260] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1815a7718a854d438ac90252a8d971b5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 40477 } } peers { permanent_uuid: "244f104a414e4a93b54c7e093b64e712" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35447 } } peers { permanent_uuid: "3fe9a8212be143e889dea01a0c94e2d0" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44399 } }
I20260806 22:46:03.355943 18977 heartbeater.cc:503] Master 127.23.100.190:37783 was elected leader, sending a full tablet report...
I20260806 22:46:03.357184 19011 ts_tablet_manager.cc:1428] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: Time spent starting tablet: real 0.019s user 0.017s sys 0.001s
I20260806 22:46:03.358268 18780 heartbeater.cc:503] Master 127.23.100.190:37783 was elected leader, sending a full tablet report...
I20260806 22:46:03.358798 19012 ts_tablet_manager.cc:1428] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0: Time spent starting tablet: real 0.018s user 0.015s sys 0.002s
I20260806 22:46:03.422498 19016 raft_consensus.cc:482] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:46:03.422868 19016 raft_consensus.cc:504] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1815a7718a854d438ac90252a8d971b5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 40477 } } peers { permanent_uuid: "244f104a414e4a93b54c7e093b64e712" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35447 } } peers { permanent_uuid: "3fe9a8212be143e889dea01a0c94e2d0" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44399 } }
I20260806 22:46:03.424642 19016 leader_election.cc:288] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244f104a414e4a93b54c7e093b64e712 (127.23.100.131:35447), 3fe9a8212be143e889dea01a0c94e2d0 (127.23.100.130:44399)
I20260806 22:46:03.434187 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.425212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:51777 (local address 127.23.100.130:44399)
0806 22:46:03.425406 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:03.425414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:03.426592 (+ 1178us) server_negotiation.cc:407] Connection header received
0806 22:46:03.426763 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:03.426767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:03.426855 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:03.426958 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:03.427614 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.428618 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.431805 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.433049 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.433222 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.433295 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:03.433798 (+ 503us) server_negotiation.cc:299] Negotiation successful
0806 22:46:03.433936 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 22:46:03.434187 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.424978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:56165 (local address 127.23.100.131:35447)
0806 22:46:03.425198 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:03.425207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:03.426217 (+ 1010us) server_negotiation.cc:407] Connection header received
0806 22:46:03.426519 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:03.426523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:03.426622 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:03.426732 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:03.427486 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.428591 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.431814 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.433102 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.433244 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.433307 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:03.433809 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 22:46:03.433936 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:46:03.434374 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.424905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:35447 (local address 127.23.100.129:56165)
0806 22:46:03.426091 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:03.426116 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:03.426352 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:03.426783 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:03.426787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:03.426820 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:03.427307 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.427320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.428761 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.428767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.431566 (+ 2799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.431575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.433312 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.433318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.433583 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.433648 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 22:46:03.433750 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 22:46:03.433908 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":228,"thread_start_us":1202,"threads_started":1}
I20260806 22:46:03.434392 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.425112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:44399 (local address 127.23.100.129:51777)
0806 22:46:03.426462 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:03.426489 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:03.426624 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:03.426975 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:03.426979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:03.427003 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:03.427434 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.427448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.428762 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.428767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.431556 (+ 2789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:03.431574 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.433291 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.433299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:03.433583 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.433649 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 22:46:03.433750 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 22:46:03.433908 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"mutex_wait_us":313,"thread_start_us":613,"threads_started":1}
I20260806 22:46:03.435521 18919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196a7d893e6e4c07b317d9c38fd9c6a8" candidate_uuid: "1815a7718a854d438ac90252a8d971b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244f104a414e4a93b54c7e093b64e712" is_pre_election: true
I20260806 22:46:03.435837 18709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196a7d893e6e4c07b317d9c38fd9c6a8" candidate_uuid: "1815a7718a854d438ac90252a8d971b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe9a8212be143e889dea01a0c94e2d0" is_pre_election: true
I20260806 22:46:03.436092 18919 raft_consensus.cc:2454] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1815a7718a854d438ac90252a8d971b5 in term 0.
I20260806 22:46:03.436404 18709 raft_consensus.cc:2454] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1815a7718a854d438ac90252a8d971b5 in term 0.
I20260806 22:46:03.436892 18394 leader_election.cc:302] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1815a7718a854d438ac90252a8d971b5, 244f104a414e4a93b54c7e093b64e712; no voters:
I20260806 22:46:03.437351 19016 raft_consensus.cc:2781] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:46:03.437534 19016 raft_consensus.cc:482] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:46:03.437674 19016 raft_consensus.cc:3037] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:03.439885 19016 raft_consensus.cc:504] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1815a7718a854d438ac90252a8d971b5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 40477 } } peers { permanent_uuid: "244f104a414e4a93b54c7e093b64e712" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35447 } } peers { permanent_uuid: "3fe9a8212be143e889dea01a0c94e2d0" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44399 } }
I20260806 22:46:03.440575 19016 leader_election.cc:288] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [CANDIDATE]: Term 1 election: Requested vote from peers 244f104a414e4a93b54c7e093b64e712 (127.23.100.131:35447), 3fe9a8212be143e889dea01a0c94e2d0 (127.23.100.130:44399)
I20260806 22:46:03.441075 18919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196a7d893e6e4c07b317d9c38fd9c6a8" candidate_uuid: "1815a7718a854d438ac90252a8d971b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244f104a414e4a93b54c7e093b64e712"
I20260806 22:46:03.441075 18709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196a7d893e6e4c07b317d9c38fd9c6a8" candidate_uuid: "1815a7718a854d438ac90252a8d971b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe9a8212be143e889dea01a0c94e2d0"
I20260806 22:46:03.441291 18709 raft_consensus.cc:3037] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:03.441291 18919 raft_consensus.cc:3037] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:03.444739 18919 raft_consensus.cc:2454] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1815a7718a854d438ac90252a8d971b5 in term 1.
I20260806 22:46:03.444739 18709 raft_consensus.cc:2454] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1815a7718a854d438ac90252a8d971b5 in term 1.
I20260806 22:46:03.445231 18397 leader_election.cc:302] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1815a7718a854d438ac90252a8d971b5, 3fe9a8212be143e889dea01a0c94e2d0; no voters:
I20260806 22:46:03.445662 19016 raft_consensus.cc:2781] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:03.446165 19016 raft_consensus.cc:686] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 1 LEADER]: Becoming Leader. State: Replica: 1815a7718a854d438ac90252a8d971b5, State: Running, Role: LEADER
I20260806 22:46:03.446669 19016 consensus_queue.cc:237] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1815a7718a854d438ac90252a8d971b5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 40477 } } peers { permanent_uuid: "244f104a414e4a93b54c7e093b64e712" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35447 } } peers { permanent_uuid: "3fe9a8212be143e889dea01a0c94e2d0" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44399 } }
I20260806 22:46:03.451565 18280 catalog_manager.cc:5495] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1815a7718a854d438ac90252a8d971b5 (127.23.100.129). New cstate: current_term: 1 leader_uuid: "1815a7718a854d438ac90252a8d971b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1815a7718a854d438ac90252a8d971b5" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 40477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "244f104a414e4a93b54c7e093b64e712" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fe9a8212be143e889dea01a0c94e2d0" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44399 } health_report { overall_health: UNKNOWN } } }
I20260806 22:46:03.468602 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.462880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.23.100.129:40477)
0806 22:46:03.463099 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:03.463105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:03.463219 (+ 114us) server_negotiation.cc:407] Connection header received
0806 22:46:03.463461 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:03.463465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:03.463579 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:03.463740 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:46:03.464170 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.464866 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.466483 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.466916 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.467055 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.467560 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:46:03.468032 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:46:03.468186 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:03.468266 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 22:46:03.468347 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 22:46:03.477016 18709 raft_consensus.cc:1261] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0 [term 1 FOLLOWER]: Refusing update from remote peer 1815a7718a854d438ac90252a8d971b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:03.477020 18919 raft_consensus.cc:1261] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712 [term 1 FOLLOWER]: Refusing update from remote peer 1815a7718a854d438ac90252a8d971b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:03.478008 19025 consensus_queue.cc:1035] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe9a8212be143e889dea01a0c94e2d0" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 44399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:46:03.478386 19016 consensus_queue.cc:1035] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244f104a414e4a93b54c7e093b64e712" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 22:46:03.492288 18978 tablet.cc:2369] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:46:03.493821 19030 mvcc.cc:205] Tried to move back new op lower bound from 7315686864792182784 to 7315686864681185280. Current Snapshot: MvccSnapshot[applied={T|T < 7315686864792182784}]
I20260806 22:46:03.505060 18280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:03.505334 18280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:03.508942 18280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ce889ee9d3544a78de984398438f383: Sending DeleteTablet for 3 replicas of tablet 196a7d893e6e4c07b317d9c38fd9c6a8
I20260806 22:46:03.510143 18888 tablet_service.cc:1514] Processing DeleteTablet for tablet 196a7d893e6e4c07b317d9c38fd9c6a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:03 UTC) from {username='slave'} at 127.0.0.1:32884
I20260806 22:46:03.510143 18469 tablet_service.cc:1514] Processing DeleteTablet for tablet 196a7d893e6e4c07b317d9c38fd9c6a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:03 UTC) from {username='slave'} at 127.0.0.1:34432
I20260806 22:46:03.510280 18674 tablet_service.cc:1514] Processing DeleteTablet for tablet 196a7d893e6e4c07b317d9c38fd9c6a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:03 UTC) from {username='slave'} at 127.0.0.1:52578
I20260806 22:46:03.513374 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 18372
I20260806 22:46:03.525426 19044 tablet_replica.cc:330] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5: stopping tablet replica
I20260806 22:46:03.525992 19044 raft_consensus.cc:2229] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:46:03.526751 19044 raft_consensus.cc:2258] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:03.529220 19044 ts_tablet_manager.cc:1905] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:03.532552 19046 tablet_replica.cc:330] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0: stopping tablet replica
I20260806 22:46:03.532991 19046 raft_consensus.cc:2229] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:03.533571 19046 raft_consensus.cc:2258] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:03.534415 19045 tablet_replica.cc:330] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: stopping tablet replica
I20260806 22:46:03.534842 19045 raft_consensus.cc:2229] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:03.535317 19045 raft_consensus.cc:2258] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:03.537695 19044 ts_tablet_manager.cc:1918] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:03.537894 19044 log.cc:1196] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/196a7d893e6e4c07b317d9c38fd9c6a8
I20260806 22:46:03.537914 19045 ts_tablet_manager.cc:1905] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:03.538084 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.514755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.23.100.129:40477)
0806 22:46:03.514987 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:03.514996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:03.520717 (+ 5721us) server_negotiation.cc:407] Connection header received
0806 22:46:03.521813 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:03.521822 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:03.521962 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:03.522069 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:03.523584 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.525149 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.529678 (+ 4529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.530324 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.530426 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.536145 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:03.536158 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:03.536176 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:03.537001 (+ 825us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:03.537070 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:03.537097 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:03.537220 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:03.537454 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 22:46:03.537598 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 22:46:03.539285 19044 ts_tablet_manager.cc:1939] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 1815a7718a854d438ac90252a8d971b5: Deleting consensus metadata
I20260806 22:46:03.539577 18559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:03.542798 19046 ts_tablet_manager.cc:1905] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:03.546229 19045 ts_tablet_manager.cc:1918] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:03.546418 19045 log.cc:1196] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/196a7d893e6e4c07b317d9c38fd9c6a8
I20260806 22:46:03.546981 19045 ts_tablet_manager.cc:1939] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 244f104a414e4a93b54c7e093b64e712: Deleting consensus metadata
I20260806 22:46:03.548372 19046 ts_tablet_manager.cc:1918] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:03.548569 19046 log.cc:1196] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/196a7d893e6e4c07b317d9c38fd9c6a8
I20260806 22:46:03.548998 18255 catalog_manager.cc:4841] TS 244f104a414e4a93b54c7e093b64e712 (127.23.100.131:35447): tablet 196a7d893e6e4c07b317d9c38fd9c6a8 (table test-table [id=5b8867bb845d4d4fa791a85f39e1a647]) successfully deleted
I20260806 22:46:03.549167 19046 ts_tablet_manager.cc:1939] T 196a7d893e6e4c07b317d9c38fd9c6a8 P 3fe9a8212be143e889dea01a0c94e2d0: Deleting consensus metadata
I20260806 22:46:03.550853 18259 catalog_manager.cc:4841] TS 3fe9a8212be143e889dea01a0c94e2d0 (127.23.100.130:44399): tablet 196a7d893e6e4c07b317d9c38fd9c6a8 (table test-table [id=5b8867bb845d4d4fa791a85f39e1a647]) successfully deleted
I20260806 22:46:03.642105 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 18372
I20260806 22:46:03.642374 18260 catalog_manager.cc:4841] TS 1815a7718a854d438ac90252a8d971b5 (127.23.100.129:40477): tablet 196a7d893e6e4c07b317d9c38fd9c6a8 (table test-table [id=5b8867bb845d4d4fa791a85f39e1a647]) successfully deleted
I20260806 22:46:03.666186 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 18578
I20260806 22:46:03.673468 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.667014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.23.100.130:44399)
0806 22:46:03.667209 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:03.667216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:03.667317 (+ 101us) server_negotiation.cc:407] Connection header received
0806 22:46:03.667558 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:03.667563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:03.667663 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:03.667781 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:03.668217 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.669003 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.670402 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.670812 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.670945 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.672081 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:03.672090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:03.672103 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:03.672680 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:03.672733 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:03.672752 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:03.672879 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:03.673022 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 22:46:03.673129 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 22:46:03.674268 18773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:03.774889 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 18578
I20260806 22:46:03.799438 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 18784
I20260806 22:46:03.806715 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:03.800271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.23.100.131:35447)
0806 22:46:03.800453 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:03.800458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:03.800478 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:46:03.800706 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:03.800711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:03.800822 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:03.800923 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:03.801309 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.802123 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.803451 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:03.803830 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:03.803928 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:03.805050 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:03.805058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:03.805072 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:03.805751 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:03.805800 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:03.805828 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:03.805920 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:03.806121 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 22:46:03.806354 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 22:46:03.807379 18971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:03.899232 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 18784
I20260806 22:46:03.924364 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 18232
I20260806 22:46:03.925616 18325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:04.043650 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 18232
2026-08-06T22:46:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2889 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:04 dist-test-slave-zr1t krb5kdc[19061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:04 dist-test-slave-zr1t krb5kdc[19061](info): set up 2 sockets
Aug 06 22:46:04 dist-test-slave-zr1t krb5kdc[19061](info): commencing operation
krb5kdc: starting...
W20260806 22:46:05.384166 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:05 dist-test-slave-zr1t krb5kdc[19061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:46:05.694591 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33093
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:33093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:06.034643 19083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:06.035005 19083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:06.035171 19083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:06.042474 19083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:06.042621 19083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:06.042732 19083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:06.042892 19083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:06.042968 19083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:06.052654 19083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:33093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:06.053900 19083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:06.055111 19083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:06.066888 19088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:06.066834 19089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:06.066932 19092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:06.068039 19083 server_base.cc:1034] running on GCE node
I20260806 22:46:06.068801 19083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:06.070247 19083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:06.071484 19083 hybrid_clock.cc:648] HybridClock initialized: now 1786056366071445 us; error 44 us; skew 500 ppm
I20260806 22:46:06.071823 19083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:06 dist-test-slave-zr1t krb5kdc[19061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056366, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:06.088630 19083 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:46:06.090250 19083 webserver.cc:460] Webserver started at http://127.23.100.190:44137/ using document root <none> and password file <none>
I20260806 22:46:06.090977 19083 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:06.091140 19083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:06.091452 19083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:06.094483 19083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "eb3ae7f2de1d4490b50facb4630d706e"
format_stamp: "Formatted at 2026-08-06 22:46:06 on dist-test-slave-zr1t"
server_key: "5d6d49162f6bf4fb587f749aa4ae751b"
server_key_iv: "16d176da928f8cea3063875c409fc0bd"
server_key_version: "encryptionkey@0"
I20260806 22:46:06.095244 19083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "eb3ae7f2de1d4490b50facb4630d706e"
format_stamp: "Formatted at 2026-08-06 22:46:06 on dist-test-slave-zr1t"
server_key: "5d6d49162f6bf4fb587f749aa4ae751b"
server_key_iv: "16d176da928f8cea3063875c409fc0bd"
server_key_version: "encryptionkey@0"
I20260806 22:46:06.100247 19083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:46:06.103569 19101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:06.104795 19083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:46:06.104996 19083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "eb3ae7f2de1d4490b50facb4630d706e"
format_stamp: "Formatted at 2026-08-06 22:46:06 on dist-test-slave-zr1t"
server_key: "5d6d49162f6bf4fb587f749aa4ae751b"
server_key_iv: "16d176da928f8cea3063875c409fc0bd"
server_key_version: "encryptionkey@0"
I20260806 22:46:06.105193 19083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:06.117136 19083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:06.121095 19083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:06.121392 19083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:06.140059 19083 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:33093
I20260806 22:46:06.140082 19184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:33093 every 8 connection(s)
I20260806 22:46:06.141417 19083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:06.143306 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 19083
I20260806 22:46:06.143527 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:06.144001 23954 external_mini_cluster.cc:1428] Setting key 7747633c0541ded172555eb08e845f31
I20260806 22:46:06.145572 19185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 22:46:06 dist-test-slave-zr1t krb5kdc[19061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:46:06.159340 19185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e: Bootstrap starting.
I20260806 22:46:06.163410 19185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:06.164798 19185 log.cc:824] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:06.168392 19185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e: No bootstrap required, opened a new log
I20260806 22:46:06.169731 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:06.145891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.23.100.190:33093)
0806 22:46:06.146702 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:06.146719 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:06.146770 (+ 51us) server_negotiation.cc:407] Connection header received
0806 22:46:06.148783 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:06.148810 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:06.149288 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:06.149749 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:06.151097 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:06.153170 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:06.154641 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:06.155723 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:06.155927 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:06.160556 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:06.160568 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:06.160587 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:06.160638 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:06.164852 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:06.165830 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:06.165837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:06.165843 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:06.166014 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:06.166528 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:06.166532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:06.166535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:06.167203 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:06.167541 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:06.167907 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 22:46:06.168339 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":311,"threads_started":1}
I20260806 22:46:06.177224 19185 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3ae7f2de1d4490b50facb4630d706e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33093 } }
I20260806 22:46:06.177616 19185 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:06.177713 19185 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3ae7f2de1d4490b50facb4630d706e, State: Initialized, Role: FOLLOWER
I20260806 22:46:06.178262 19185 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3ae7f2de1d4490b50facb4630d706e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33093 } }
I20260806 22:46:06.178468 19185 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:06.178566 19185 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:06.178694 19185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:06.180958 19185 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3ae7f2de1d4490b50facb4630d706e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33093 } }
I20260806 22:46:06.181444 19185 leader_election.cc:302] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb3ae7f2de1d4490b50facb4630d706e; no voters:
I20260806 22:46:06.182086 19185 leader_election.cc:288] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:06.182261 19192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:06.183487 19192 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [term 1 LEADER]: Becoming Leader. State: Replica: eb3ae7f2de1d4490b50facb4630d706e, State: Running, Role: LEADER
I20260806 22:46:06.184114 19192 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3ae7f2de1d4490b50facb4630d706e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33093 } }
I20260806 22:46:06.184331 19185 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:06.187448 19193 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb3ae7f2de1d4490b50facb4630d706e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3ae7f2de1d4490b50facb4630d706e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33093 } } }
I20260806 22:46:06.187866 19193 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:06.187574 19194 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb3ae7f2de1d4490b50facb4630d706e. Latest consensus state: current_term: 1 leader_uuid: "eb3ae7f2de1d4490b50facb4630d706e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3ae7f2de1d4490b50facb4630d706e" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33093 } } }
I20260806 22:46:06.187997 19194 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:06.191290 19208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:06.196493 19208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:06.206694 19208 catalog_manager.cc:1347] Generated new cluster ID: 3f9fcb5c613e4cb5a4010454ba1e705b
I20260806 22:46:06.206840 19208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:06.222627 19208 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:06.224038 19208 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:06.231856 19208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e: Generated new TSK 0
I20260806 22:46:06.232769 19208 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:46:06.312572 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33093
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:06.657315 19226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:06.657680 19226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:06.657873 19226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:06.665191 19226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:06.665293 19226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:06.665490 19226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:46:06.676688 19226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:06.678054 19226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:06.679387 19226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:06.693410 19234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:06.693499 19235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:06.693507 19238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:06.694659 19226 server_base.cc:1034] running on GCE node
I20260806 22:46:06.695309 19226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:06.696244 19226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:06.697440 19226 hybrid_clock.cc:648] HybridClock initialized: now 1786056366697375 us; error 56 us; skew 500 ppm
I20260806 22:46:06.697834 19226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:06 dist-test-slave-zr1t krb5kdc[19061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056366, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:06.700955 19226 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:46:06.702384 19226 webserver.cc:460] Webserver started at http://127.23.100.129:38929/ using document root <none> and password file <none>
I20260806 22:46:06.703029 19226 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:06.703136 19226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:06.703397 19226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:06.706233 19226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "64d3810c5f0e49239ce3bbde68f83cfe"
format_stamp: "Formatted at 2026-08-06 22:46:06 on dist-test-slave-zr1t"
server_key: "5c917f92fca4ef29d62ae8bed0af224b"
server_key_iv: "4db611793372ace3b4df38ff47f6e69e"
server_key_version: "encryptionkey@0"
I20260806 22:46:06.706949 19226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "64d3810c5f0e49239ce3bbde68f83cfe"
format_stamp: "Formatted at 2026-08-06 22:46:06 on dist-test-slave-zr1t"
server_key: "5c917f92fca4ef29d62ae8bed0af224b"
server_key_iv: "4db611793372ace3b4df38ff47f6e69e"
server_key_version: "encryptionkey@0"
I20260806 22:46:06.712257 19226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:46:06.715799 19251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:06.717193 19226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:46:06.717365 19226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "64d3810c5f0e49239ce3bbde68f83cfe"
format_stamp: "Formatted at 2026-08-06 22:46:06 on dist-test-slave-zr1t"
server_key: "5c917f92fca4ef29d62ae8bed0af224b"
server_key_iv: "4db611793372ace3b4df38ff47f6e69e"
server_key_version: "encryptionkey@0"
I20260806 22:46:06.717651 19226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:06.732880 19226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:06.737112 19226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:06.737495 19226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:06.738487 19226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:06.739848 19226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:06.739933 19226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:06.740108 19226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:06.740190 19226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:06.775534 19226 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:33205
I20260806 22:46:06.775576 19419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:33205 every 8 connection(s)
I20260806 22:46:06.776942 19226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:46:06.785676 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 19226
I20260806 22:46:06.785882 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:46:06.786329 23954 external_mini_cluster.cc:1428] Setting key 76bb55b8d68ec503fc00c294fa850861
Aug 06 22:46:06 dist-test-slave-zr1t krb5kdc[19061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056366, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:46:06.799363 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:06.779631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:58043 (local address 127.23.100.190:33093)
0806 22:46:06.780179 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:06.780185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:06.780951 (+ 766us) server_negotiation.cc:407] Connection header received
0806 22:46:06.782850 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:06.782855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:06.782938 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:06.783068 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:06.784539 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:06.785306 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:06.787268 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:06.787864 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:06.788003 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:06.792344 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:06.792350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:06.792354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:06.792417 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:06.795600 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:06.796594 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:06.796602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:06.796607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:06.796743 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:06.797255 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:06.797261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:06.797267 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:06.797644 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:06.797779 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:06.798703 (+ 924us) server_negotiation.cc:299] Negotiation successful
0806 22:46:06.798938 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":306,"threads_started":1}
I20260806 22:46:06.801007 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:06.779995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33093 (local address 127.23.100.129:58043)
0806 22:46:06.780776 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:06.780821 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:06.782589 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:06.783301 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:06.783309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:06.783800 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:06.784360 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:06.784377 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:06.785462 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:06.785467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:06.787128 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:06.787141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:06.788110 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:06.788118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:06.788423 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:06.789159 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:46:06.789192 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:06.792109 (+ 2917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:06.795779 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:06.795791 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:06.795811 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:06.796460 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:06.796854 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:06.796863 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:06.796869 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:06.797132 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:06.797787 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:06.797799 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:06.798276 (+ 477us) client_negotiation.cc:769] Sending connection context
0806 22:46:06.798587 (+ 311us) client_negotiation.cc:240] Negotiation successful
0806 22:46:06.799186 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":317,"threads_started":1}
I20260806 22:46:06.802983 19421 heartbeater.cc:347] Connected to a master server at 127.23.100.190:33093
I20260806 22:46:06.803318 19421 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:06.804134 19421 heartbeater.cc:511] Master 127.23.100.190:33093 requested a full tablet report, sending...
I20260806 22:46:06.806389 19123 ts_manager.cc:194] Registered new tserver with Master: 64d3810c5f0e49239ce3bbde68f83cfe (127.23.100.129:33205)
I20260806 22:46:06.808569 19123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:58043
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:46:06.861110 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33093
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:07.219462 19428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:07.219748 19428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:07.219915 19428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:07.227537 19428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:07.227655 19428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:07.227854 19428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:46:07.237828 19428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:07.239100 19428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:07.240558 19428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:07.254022 19434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:07.254073 19438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:07.253975 19435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:07.254937 19428 server_base.cc:1034] running on GCE node
I20260806 22:46:07.255664 19428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:07.256661 19428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:07.257884 19428 hybrid_clock.cc:648] HybridClock initialized: now 1786056367257831 us; error 51 us; skew 500 ppm
I20260806 22:46:07.258208 19428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:07 dist-test-slave-zr1t krb5kdc[19061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056367, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:07.261372 19428 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:46:07.263020 19428 webserver.cc:460] Webserver started at http://127.23.100.130:43077/ using document root <none> and password file <none>
I20260806 22:46:07.263852 19428 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:07.264002 19428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:07.264330 19428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:07.267323 19428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "1ecd404f090c49b4a1f8e1ca8e57c954"
format_stamp: "Formatted at 2026-08-06 22:46:07 on dist-test-slave-zr1t"
server_key: "5089092760d6ab07d28c47d696e7e7fe"
server_key_iv: "296c4fbd8cdc72d2118cadb1a2a14973"
server_key_version: "encryptionkey@0"
I20260806 22:46:07.268107 19428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "1ecd404f090c49b4a1f8e1ca8e57c954"
format_stamp: "Formatted at 2026-08-06 22:46:07 on dist-test-slave-zr1t"
server_key: "5089092760d6ab07d28c47d696e7e7fe"
server_key_iv: "296c4fbd8cdc72d2118cadb1a2a14973"
server_key_version: "encryptionkey@0"
I20260806 22:46:07.273252 19428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 22:46:07.276875 19446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:07.278266 19428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:46:07.278466 19428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "1ecd404f090c49b4a1f8e1ca8e57c954"
format_stamp: "Formatted at 2026-08-06 22:46:07 on dist-test-slave-zr1t"
server_key: "5089092760d6ab07d28c47d696e7e7fe"
server_key_iv: "296c4fbd8cdc72d2118cadb1a2a14973"
server_key_version: "encryptionkey@0"
I20260806 22:46:07.278678 19428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:07.301601 19428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:07.305716 19428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:07.306039 19428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:07.307032 19428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:07.308386 19428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:07.308463 19428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:07.308549 19428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:07.308594 19428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:07.343320 19428 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:35519
I20260806 22:46:07.343417 19626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:35519 every 8 connection(s)
I20260806 22:46:07.344703 19428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:46:07.347198 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 19428
I20260806 22:46:07.347422 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:46:07.348004 23954 external_mini_cluster.cc:1428] Setting key 7aa3230d4afc812df8a66dfcbccdcdd4
Aug 06 22:46:07 dist-test-slave-zr1t krb5kdc[19061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056367, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:46:07.364307 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:07.347283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:46161 (local address 127.23.100.190:33093)
0806 22:46:07.347820 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:07.347827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:07.349027 (+ 1200us) server_negotiation.cc:407] Connection header received
0806 22:46:07.350694 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:07.350699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:07.350791 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:07.350906 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:07.352335 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.353063 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:07.354628 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.355057 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:07.355190 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:07.358638 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:07.358643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:07.358646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:07.358677 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:07.361270 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:07.362064 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:07.362068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:07.362071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:07.362198 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:07.362638 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:07.362643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:07.362645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:07.362891 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:07.363049 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:07.363712 (+ 663us) server_negotiation.cc:299] Negotiation successful
0806 22:46:07.363935 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":271,"threads_started":1}
I20260806 22:46:07.365281 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:07.347796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33093 (local address 127.23.100.130:46161)
0806 22:46:07.348593 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:07.348647 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:07.350438 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:07.351137 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:07.351145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:07.351634 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:07.352170 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:07.352188 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.353225 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:07.353230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:07.354487 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:07.354499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.355300 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:07.355308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:07.355526 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:07.356205 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:46:07.356226 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:07.358415 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:07.361420 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:07.361429 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:07.361444 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:07.361920 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:07.362313 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:07.362318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:07.362322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:07.362512 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:07.363025 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:07.363033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:07.363350 (+ 317us) client_negotiation.cc:769] Sending connection context
0806 22:46:07.363609 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 22:46:07.364124 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":338,"threads_started":1}
I20260806 22:46:07.366897 19627 heartbeater.cc:347] Connected to a master server at 127.23.100.190:33093
I20260806 22:46:07.367262 19627 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:07.368125 19627 heartbeater.cc:511] Master 127.23.100.190:33093 requested a full tablet report, sending...
I20260806 22:46:07.369959 19123 ts_manager.cc:194] Registered new tserver with Master: 1ecd404f090c49b4a1f8e1ca8e57c954 (127.23.100.130:35519)
I20260806 22:46:07.370792 19123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:46161
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:46:07.420161 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33093
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:07.778741 19636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:07.779007 19636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:07.779174 19636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:07.787072 19636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:07.787179 19636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:07.787389 19636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:46:07.797196 19636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:33093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:07.798461 19636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:07.799753 19636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 22:46:07.811710 19421 heartbeater.cc:503] Master 127.23.100.190:33093 was elected leader, sending a full tablet report...
W20260806 22:46:07.813305 19642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:07.813344 19647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:07.813910 19644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:07.818630 19636 server_base.cc:1034] running on GCE node
I20260806 22:46:07.819216 19636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:07.820122 19636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:07.821319 19636 hybrid_clock.cc:648] HybridClock initialized: now 1786056367821278 us; error 40 us; skew 500 ppm
I20260806 22:46:07.821728 19636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:07 dist-test-slave-zr1t krb5kdc[19061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056367, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:07.824945 19636 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:46:07.826480 19636 webserver.cc:460] Webserver started at http://127.23.100.131:44165/ using document root <none> and password file <none>
I20260806 22:46:07.827136 19636 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:07.827239 19636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:07.827505 19636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:07.830389 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "36497d71e6b745b185e7d6f4eabe432f"
format_stamp: "Formatted at 2026-08-06 22:46:07 on dist-test-slave-zr1t"
server_key: "48f3678dbe08ce1710908154c523f16f"
server_key_iv: "cea9f98a1a83d45f24fc1e112f5a119e"
server_key_version: "encryptionkey@0"
I20260806 22:46:07.831099 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "36497d71e6b745b185e7d6f4eabe432f"
format_stamp: "Formatted at 2026-08-06 22:46:07 on dist-test-slave-zr1t"
server_key: "48f3678dbe08ce1710908154c523f16f"
server_key_iv: "cea9f98a1a83d45f24fc1e112f5a119e"
server_key_version: "encryptionkey@0"
I20260806 22:46:07.836194 19636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:46:07.839613 19659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:07.841053 19636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260806 22:46:07.841214 19636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "36497d71e6b745b185e7d6f4eabe432f"
format_stamp: "Formatted at 2026-08-06 22:46:07 on dist-test-slave-zr1t"
server_key: "48f3678dbe08ce1710908154c523f16f"
server_key_iv: "cea9f98a1a83d45f24fc1e112f5a119e"
server_key_version: "encryptionkey@0"
I20260806 22:46:07.841401 19636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:07.868453 19636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:07.873051 19636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:07.873374 19636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:07.874425 19636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:07.875838 19636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:07.875926 19636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:07.876101 19636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:07.876155 19636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:07.912040 19636 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:38957
I20260806 22:46:07.912101 19832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:38957 every 8 connection(s)
I20260806 22:46:07.913389 19636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:46:07.914656 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 19636
I20260806 22:46:07.914865 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:46:07.915330 23954 external_mini_cluster.cc:1428] Setting key 62d94da79422e43d3abaab7eef09db45
Aug 06 22:46:07 dist-test-slave-zr1t krb5kdc[19061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056367, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:46:07.934564 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:07.916096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:45423 (local address 127.23.100.190:33093)
0806 22:46:07.916647 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:07.916654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:07.917761 (+ 1107us) server_negotiation.cc:407] Connection header received
0806 22:46:07.919731 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:07.919737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:07.919820 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:07.919952 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:07.921553 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.922442 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:07.924027 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.924436 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:07.924567 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:07.927986 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:07.927991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:07.927994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:07.928027 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:07.931387 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:07.932125 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:07.932132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:07.932136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:07.932263 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:07.932710 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:07.932715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:07.932719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:07.932998 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:07.933167 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:07.933941 (+ 774us) server_negotiation.cc:299] Negotiation successful
0806 22:46:07.934162 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":262,"threads_started":1}
I20260806 22:46:07.935511 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:07.916654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:33093 (local address 127.23.100.131:45423)
0806 22:46:07.917628 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:07.917666 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:07.919461 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:07.920189 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:07.920197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:07.920693 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:07.921327 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:07.921345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.922605 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:07.922609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:07.923863 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:07.923876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.924671 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:07.924678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:07.924919 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:07.925594 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:46:07.925616 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:07.927766 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:07.931540 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:07.931549 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:07.931566 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:07.932003 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:07.932387 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:07.932392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:07.932395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:07.932591 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:07.933141 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:07.933149 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:07.933496 (+ 347us) client_negotiation.cc:769] Sending connection context
0806 22:46:07.933848 (+ 352us) client_negotiation.cc:240] Negotiation successful
0806 22:46:07.934404 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":328,"threads_started":1}
I20260806 22:46:07.936928 19833 heartbeater.cc:347] Connected to a master server at 127.23.100.190:33093
I20260806 22:46:07.937237 19833 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:07.938014 19833 heartbeater.cc:511] Master 127.23.100.190:33093 requested a full tablet report, sending...
I20260806 22:46:07.939415 19123 ts_manager.cc:194] Registered new tserver with Master: 36497d71e6b745b185e7d6f4eabe432f (127.23.100.131:38957)
I20260806 22:46:07.940227 19123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:45423
I20260806 22:46:07.943431 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:46:07.960585 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:07.947775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.23.100.190:33093)
0806 22:46:07.948004 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:07.948013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:07.948446 (+ 433us) server_negotiation.cc:407] Connection header received
0806 22:46:07.948573 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:07.948577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:07.948637 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:07.948714 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:07.950207 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.950965 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:07.952235 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.952626 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:07.952718 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:07.955456 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:07.955461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:07.955464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:07.955494 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:07.958058 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:07.958795 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:07.958799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:07.958801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:07.958909 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:07.959274 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:07.959277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:07.959279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:07.959567 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:07.959755 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:07.960090 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 22:46:07.960266 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 22:46:07.964437 19123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:46:07.971316 19123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:46:08.004860 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:07.983501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:33205 (local address 127.0.0.1:52350)
0806 22:46:07.993137 (+ 9636us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:07.993201 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:07.993432 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:07.995416 (+ 1984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:07.995427 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:07.995484 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:07.996091 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:07.996109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.998848 (+ 2739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:07.998857 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:08.001647 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:08.001663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.003828 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.003837 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:08.004056 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.004120 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 22:46:08.004254 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 22:46:08.004462 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9313,"thread_start_us":17107,"threads_started":1}
I20260806 22:46:08.005741 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:07.984091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.23.100.129:33205)
0806 22:46:07.994046 (+ 9955us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:07.994059 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:07.994100 (+ 41us) server_negotiation.cc:407] Connection header received
0806 22:46:07.994236 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:07.994248 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:07.994573 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:07.994838 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:07.996385 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:07.998599 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.001841 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.003458 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.003678 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.003881 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:08.005207 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0806 22:46:08.005391 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9715,"thread_start_us":10989,"threads_started":1}
I20260806 22:46:08.009912 19319 tablet_service.cc:1467] Processing CreateTablet for tablet a4619aac3b66444e95be1639c0266b02 (DEFAULT_TABLE table=test-table [id=891106578ac84f629f9c24e1e35ccf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:08.011452 19319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4619aac3b66444e95be1639c0266b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:08.011899 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:07.983501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:35519 (local address 127.0.0.1:49296)
0806 22:46:08.000381 (+ 16880us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:08.000406 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:08.000544 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:08.003070 (+ 2526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:08.003079 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:08.003121 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:08.003748 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:08.003761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.006122 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.006127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:08.008968 (+ 2841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:08.008988 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.010971 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.010980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:08.011158 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.011191 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 22:46:08.011300 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 22:46:08.011445 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16739,"mutex_wait_us":132,"thread_start_us":19884,"threads_started":1}
I20260806 22:46:08.012140 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:07.984989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:38957 (local address 127.0.0.1:42246)
0806 22:46:08.003764 (+ 18775us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:08.003783 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:08.004046 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:08.005008 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:08.005012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:08.005035 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:08.005375 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:08.005388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.007201 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.007208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:08.009398 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:08.009412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.011283 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.011292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:08.011473 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.011508 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 22:46:08.011609 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 22:46:08.011743 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18622,"mutex_wait_us":4,"thread_start_us":7650,"threads_started":1}
I20260806 22:46:08.012672 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:07.983621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.23.100.130:35519)
0806 22:46:08.002405 (+ 18784us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:08.002415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:08.002444 (+ 29us) server_negotiation.cc:407] Connection header received
0806 22:46:08.002539 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:08.002548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:08.002813 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:08.003045 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:08.004080 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.005929 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.009176 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.010740 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.010920 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.011095 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:08.012145 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0806 22:46:08.012319 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18588,"thread_start_us":18486,"threads_started":1}
I20260806 22:46:08.012960 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:07.985295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.23.100.131:38957)
0806 22:46:07.996586 (+ 11291us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:07.996598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:08.003904 (+ 7306us) server_negotiation.cc:407] Connection header received
0806 22:46:08.004256 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:08.004264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:08.004499 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:08.004663 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:08.005628 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.007052 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.009600 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.011082 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.011283 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.011450 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:08.012441 (+ 991us) server_negotiation.cc:299] Negotiation successful
0806 22:46:08.012612 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11071,"thread_start_us":10911,"threads_started":1}
I20260806 22:46:08.016021 19736 tablet_service.cc:1467] Processing CreateTablet for tablet a4619aac3b66444e95be1639c0266b02 (DEFAULT_TABLE table=test-table [id=891106578ac84f629f9c24e1e35ccf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:08.017251 19736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4619aac3b66444e95be1639c0266b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:08.019971 19519 tablet_service.cc:1467] Processing CreateTablet for tablet a4619aac3b66444e95be1639c0266b02 (DEFAULT_TABLE table=test-table [id=891106578ac84f629f9c24e1e35ccf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:08.021426 19519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4619aac3b66444e95be1639c0266b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:08.025061 19861 tablet_bootstrap.cc:492] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe: Bootstrap starting.
I20260806 22:46:08.028925 19861 tablet_bootstrap.cc:654] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:08.030531 19861 log.cc:824] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:08.032130 19862 tablet_bootstrap.cc:492] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f: Bootstrap starting.
I20260806 22:46:08.033836 19861 tablet_bootstrap.cc:492] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe: No bootstrap required, opened a new log
I20260806 22:46:08.034113 19861 ts_tablet_manager.cc:1397] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 22:46:08.035567 19862 tablet_bootstrap.cc:654] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:08.036239 19865 tablet_bootstrap.cc:492] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: Bootstrap starting.
I20260806 22:46:08.037204 19862 log.cc:824] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:08.039605 19865 tablet_bootstrap.cc:654] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:08.040501 19862 tablet_bootstrap.cc:492] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f: No bootstrap required, opened a new log
I20260806 22:46:08.040788 19862 ts_tablet_manager.cc:1397] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 22:46:08.040887 19865 log.cc:824] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:08.043321 19861 raft_consensus.cc:348] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 35519 } } peers { permanent_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 33205 } } peers { permanent_uuid: "36497d71e6b745b185e7d6f4eabe432f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38957 } }
I20260806 22:46:08.043763 19861 raft_consensus.cc:374] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:08.043860 19861 raft_consensus.cc:729] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d3810c5f0e49239ce3bbde68f83cfe, State: Initialized, Role: FOLLOWER
I20260806 22:46:08.044459 19865 tablet_bootstrap.cc:492] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: No bootstrap required, opened a new log
I20260806 22:46:08.044775 19865 ts_tablet_manager.cc:1397] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260806 22:46:08.044618 19861 consensus_queue.cc:260] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 35519 } } peers { permanent_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 33205 } } peers { permanent_uuid: "36497d71e6b745b185e7d6f4eabe432f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38957 } }
I20260806 22:46:08.046463 19861 ts_tablet_manager.cc:1428] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260806 22:46:08.051448 19862 raft_consensus.cc:348] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 35519 } } peers { permanent_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 33205 } } peers { permanent_uuid: "36497d71e6b745b185e7d6f4eabe432f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38957 } }
I20260806 22:46:08.051869 19862 raft_consensus.cc:374] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:08.051980 19862 raft_consensus.cc:729] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36497d71e6b745b185e7d6f4eabe432f, State: Initialized, Role: FOLLOWER
I20260806 22:46:08.052686 19862 consensus_queue.cc:260] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 35519 } } peers { permanent_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 33205 } } peers { permanent_uuid: "36497d71e6b745b185e7d6f4eabe432f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38957 } }
I20260806 22:46:08.053929 19833 heartbeater.cc:503] Master 127.23.100.190:33093 was elected leader, sending a full tablet report...
I20260806 22:46:08.054636 19862 ts_tablet_manager.cc:1428] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 22:46:08.054499 19865 raft_consensus.cc:348] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 35519 } } peers { permanent_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 33205 } } peers { permanent_uuid: "36497d71e6b745b185e7d6f4eabe432f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38957 } }
I20260806 22:46:08.054842 19865 raft_consensus.cc:374] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:08.054919 19865 raft_consensus.cc:729] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ecd404f090c49b4a1f8e1ca8e57c954, State: Initialized, Role: FOLLOWER
I20260806 22:46:08.055595 19865 consensus_queue.cc:260] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 35519 } } peers { permanent_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 33205 } } peers { permanent_uuid: "36497d71e6b745b185e7d6f4eabe432f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38957 } }
I20260806 22:46:08.056676 19627 heartbeater.cc:503] Master 127.23.100.190:33093 was elected leader, sending a full tablet report...
I20260806 22:46:08.057474 19865 ts_tablet_manager.cc:1428] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
W20260806 22:46:08.096752 19630 tablet.cc:2369] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:46:08.097894 19871 raft_consensus.cc:482] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:46:08.098176 19871 raft_consensus.cc:504] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 35519 } } peers { permanent_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 33205 } } peers { permanent_uuid: "36497d71e6b745b185e7d6f4eabe432f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38957 } }
I20260806 22:46:08.099835 19871 leader_election.cc:288] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ecd404f090c49b4a1f8e1ca8e57c954 (127.23.100.130:35519), 64d3810c5f0e49239ce3bbde68f83cfe (127.23.100.129:33205)
I20260806 22:46:08.106148 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:08.100167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:48359 (local address 127.23.100.130:35519)
0806 22:46:08.100414 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:08.100423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:08.100568 (+ 145us) server_negotiation.cc:407] Connection header received
0806 22:46:08.100868 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:08.100874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:08.100956 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:08.101051 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:08.101573 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.102576 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.104218 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.105134 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.105293 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.105363 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:08.105719 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 22:46:08.105873 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 22:46:08.106174 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:08.100257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:35519 (local address 127.23.100.131:48359)
0806 22:46:08.100468 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:08.100491 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:08.100636 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:08.101188 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:08.101191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:08.101215 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:08.101414 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:08.101423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.102699 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.102702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:08.104040 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:08.104061 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.105286 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.105292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:08.105492 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.105561 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 22:46:08.105669 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 22:46:08.105799 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":4,"spinlock_wait_cycles":15744}
I20260806 22:46:08.107048 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:08.100825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:50053 (local address 127.23.100.129:33205)
0806 22:46:08.101049 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:08.101055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:08.101200 (+ 145us) server_negotiation.cc:407] Connection header received
0806 22:46:08.101417 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:08.101421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:08.101506 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:08.101617 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:08.102094 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.102997 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.104708 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.105756 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.105896 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.105972 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:08.106684 (+ 712us) server_negotiation.cc:299] Negotiation successful
0806 22:46:08.106811 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 22:46:08.107117 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:08.100310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:33205 (local address 127.23.100.131:50053)
0806 22:46:08.101081 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:08.101099 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:08.101256 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:08.101621 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:08.101623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:08.101642 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:08.101946 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:08.101954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.103150 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.103157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:08.104550 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:08.104562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.106148 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.106154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:08.106515 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.106556 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 22:46:08.106657 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 22:46:08.106781 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":451,"threads_started":1}
I20260806 22:46:08.107206 19557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4619aac3b66444e95be1639c0266b02" candidate_uuid: "36497d71e6b745b185e7d6f4eabe432f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" is_pre_election: true
I20260806 22:46:08.107821 19557 raft_consensus.cc:2454] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36497d71e6b745b185e7d6f4eabe432f in term 0.
I20260806 22:46:08.108295 19348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4619aac3b66444e95be1639c0266b02" candidate_uuid: "36497d71e6b745b185e7d6f4eabe432f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" is_pre_election: true
I20260806 22:46:08.108583 19663 leader_election.cc:302] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ecd404f090c49b4a1f8e1ca8e57c954, 36497d71e6b745b185e7d6f4eabe432f; no voters:
I20260806 22:46:08.108852 19348 raft_consensus.cc:2454] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36497d71e6b745b185e7d6f4eabe432f in term 0.
I20260806 22:46:08.109028 19871 raft_consensus.cc:2781] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:46:08.109179 19871 raft_consensus.cc:482] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:46:08.109292 19871 raft_consensus.cc:3037] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:08.111755 19871 raft_consensus.cc:504] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 35519 } } peers { permanent_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 33205 } } peers { permanent_uuid: "36497d71e6b745b185e7d6f4eabe432f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38957 } }
I20260806 22:46:08.112507 19871 leader_election.cc:288] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [CANDIDATE]: Term 1 election: Requested vote from peers 1ecd404f090c49b4a1f8e1ca8e57c954 (127.23.100.130:35519), 64d3810c5f0e49239ce3bbde68f83cfe (127.23.100.129:33205)
I20260806 22:46:08.113058 19348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4619aac3b66444e95be1639c0266b02" candidate_uuid: "36497d71e6b745b185e7d6f4eabe432f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d3810c5f0e49239ce3bbde68f83cfe"
I20260806 22:46:08.113057 19557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4619aac3b66444e95be1639c0266b02" candidate_uuid: "36497d71e6b745b185e7d6f4eabe432f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954"
I20260806 22:46:08.113288 19348 raft_consensus.cc:3037] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:08.113288 19557 raft_consensus.cc:3037] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:08.116699 19557 raft_consensus.cc:2454] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36497d71e6b745b185e7d6f4eabe432f in term 1.
I20260806 22:46:08.116699 19348 raft_consensus.cc:2454] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36497d71e6b745b185e7d6f4eabe432f in term 1.
I20260806 22:46:08.117202 19664 leader_election.cc:302] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ecd404f090c49b4a1f8e1ca8e57c954, 36497d71e6b745b185e7d6f4eabe432f, 64d3810c5f0e49239ce3bbde68f83cfe; no voters:
I20260806 22:46:08.117581 19871 raft_consensus.cc:2781] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:08.118045 19871 raft_consensus.cc:686] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 1 LEADER]: Becoming Leader. State: Replica: 36497d71e6b745b185e7d6f4eabe432f, State: Running, Role: LEADER
I20260806 22:46:08.118557 19871 consensus_queue.cc:237] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 35519 } } peers { permanent_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 33205 } } peers { permanent_uuid: "36497d71e6b745b185e7d6f4eabe432f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38957 } }
I20260806 22:46:08.123596 19123 catalog_manager.cc:5495] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f reported cstate change: term changed from 0 to 1, leader changed from <none> to 36497d71e6b745b185e7d6f4eabe432f (127.23.100.131). New cstate: current_term: 1 leader_uuid: "36497d71e6b745b185e7d6f4eabe432f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 35519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 33205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36497d71e6b745b185e7d6f4eabe432f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 38957 } health_report { overall_health: HEALTHY } } }
I20260806 22:46:08.141885 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:08.136452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.23.100.131:38957)
0806 22:46:08.136647 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:08.136652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:08.136790 (+ 138us) server_negotiation.cc:407] Connection header received
0806 22:46:08.136955 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:08.136959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:08.137046 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:08.137175 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:46:08.137556 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.138276 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.139671 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.140086 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.140224 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.140802 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:46:08.141266 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:46:08.141459 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:08.141607 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 22:46:08.141655 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 22:46:08.150947 19348 raft_consensus.cc:1261] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe [term 1 FOLLOWER]: Refusing update from remote peer 36497d71e6b745b185e7d6f4eabe432f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:08.151111 19557 raft_consensus.cc:1261] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954 [term 1 FOLLOWER]: Refusing update from remote peer 36497d71e6b745b185e7d6f4eabe432f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:08.151983 19871 consensus_queue.cc:1035] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ecd404f090c49b4a1f8e1ca8e57c954" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:46:08.152298 19876 consensus_queue.cc:1035] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d3810c5f0e49239ce3bbde68f83cfe" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 33205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:46:08.164101 19881 mvcc.cc:205] Tried to move back new op lower bound from 7315686883935186944 to 7315686883817873408. Current Snapshot: MvccSnapshot[applied={T|T < 7315686883935186944}]
I20260806 22:46:08.182039 19123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:08.182369 19123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:08.186280 19123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb3ae7f2de1d4490b50facb4630d706e: Sending DeleteTablet for 3 replicas of tablet a4619aac3b66444e95be1639c0266b02
I20260806 22:46:08.187453 19519 tablet_service.cc:1514] Processing DeleteTablet for tablet a4619aac3b66444e95be1639c0266b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:08 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:49296
I20260806 22:46:08.187579 19319 tablet_service.cc:1514] Processing DeleteTablet for tablet a4619aac3b66444e95be1639c0266b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:08 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:52350
I20260806 22:46:08.188220 19736 tablet_service.cc:1514] Processing DeleteTablet for tablet a4619aac3b66444e95be1639c0266b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:08 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:42246
I20260806 22:46:08.190418 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 19226
I20260806 22:46:08.199965 19901 tablet_replica.cc:330] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: stopping tablet replica
I20260806 22:46:08.200730 19901 raft_consensus.cc:2229] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:08.201578 19901 raft_consensus.cc:2258] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:08.202858 19902 tablet_replica.cc:330] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe: stopping tablet replica
I20260806 22:46:08.203284 19902 raft_consensus.cc:2229] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:08.203801 19902 raft_consensus.cc:2258] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:08.204205 19901 ts_tablet_manager.cc:1905] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:08.209797 19901 ts_tablet_manager.cc:1918] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:08.209982 19901 log.cc:1196] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/a4619aac3b66444e95be1639c0266b02
I20260806 22:46:08.210553 19901 ts_tablet_manager.cc:1939] T a4619aac3b66444e95be1639c0266b02 P 1ecd404f090c49b4a1f8e1ca8e57c954: Deleting consensus metadata
I20260806 22:46:08.211222 19902 ts_tablet_manager.cc:1905] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:08.212428 19103 catalog_manager.cc:4841] TS 1ecd404f090c49b4a1f8e1ca8e57c954 (127.23.100.130:35519): tablet a4619aac3b66444e95be1639c0266b02 (table test-table [id=891106578ac84f629f9c24e1e35ccf35]) successfully deleted
I20260806 22:46:08.215735 19903 tablet_replica.cc:330] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f: stopping tablet replica
I20260806 22:46:08.216207 19903 raft_consensus.cc:2229] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:46:08.216938 19903 raft_consensus.cc:2258] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:08.219389 19902 ts_tablet_manager.cc:1918] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:08.219580 19902 log.cc:1196] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/a4619aac3b66444e95be1639c0266b02
I20260806 22:46:08.220142 19902 ts_tablet_manager.cc:1939] T a4619aac3b66444e95be1639c0266b02 P 64d3810c5f0e49239ce3bbde68f83cfe: Deleting consensus metadata
I20260806 22:46:08.222147 19104 catalog_manager.cc:4841] TS 64d3810c5f0e49239ce3bbde68f83cfe (127.23.100.129:33205): tablet a4619aac3b66444e95be1639c0266b02 (table test-table [id=891106578ac84f629f9c24e1e35ccf35]) successfully deleted
I20260806 22:46:08.223683 19903 ts_tablet_manager.cc:1905] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 22:46:08 dist-test-slave-zr1t krb5kdc[19061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:46:08.230736 19903 ts_tablet_manager.cc:1918] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:08.230980 19903 log.cc:1196] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/a4619aac3b66444e95be1639c0266b02
I20260806 22:46:08.231545 19903 ts_tablet_manager.cc:1939] T a4619aac3b66444e95be1639c0266b02 P 36497d71e6b745b185e7d6f4eabe432f: Deleting consensus metadata
I20260806 22:46:08.233209 19105 catalog_manager.cc:4841] TS 36497d71e6b745b185e7d6f4eabe432f (127.23.100.131:38957): tablet a4619aac3b66444e95be1639c0266b02 (table test-table [id=891106578ac84f629f9c24e1e35ccf35]) successfully deleted
I20260806 22:46:08.233796 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:08.191775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.23.100.129:33205)
0806 22:46:08.191967 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:08.191973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:08.214111 (+ 22138us) server_negotiation.cc:407] Connection header received
0806 22:46:08.214302 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:08.214308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:08.214395 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:08.214527 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:08.216301 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.217210 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.218713 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.219160 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.219326 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.225346 (+ 6020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:08.225362 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:08.225389 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:08.225438 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:08.229461 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:08.230476 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:08.230488 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:08.230491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:08.230673 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:08.231163 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:08.231169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:08.231174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:08.231690 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:08.231991 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:08.232647 (+ 656us) server_negotiation.cc:299] Negotiation successful
0806 22:46:08.232885 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 22:46:08.234484 19414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:08.325778 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 19226
I20260806 22:46:08.357126 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 19428
Aug 06 22:46:08 dist-test-slave-zr1t krb5kdc[19061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:46:08.376942 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:08.358018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.23.100.130:35519)
0806 22:46:08.358230 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:08.358237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:08.358402 (+ 165us) server_negotiation.cc:407] Connection header received
0806 22:46:08.358538 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:08.358544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:08.358623 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:08.358727 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:08.360688 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.361659 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.363800 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.364241 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.364343 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.370860 (+ 6517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:08.370881 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:08.370896 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:08.370933 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:08.373992 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:08.374742 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:08.374750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:08.374752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:08.374883 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:08.375287 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:08.375292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:08.375295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:08.375695 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:08.376013 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:08.376331 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 22:46:08.376561 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 22:46:08.377878 19619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:08.471699 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 19428
I20260806 22:46:08.498528 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 19636
Aug 06 22:46:08 dist-test-slave-zr1t krb5kdc[19061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:46:08.517452 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:08.499389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.23.100.131:38957)
0806 22:46:08.499606 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:08.499613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:08.499637 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:46:08.499828 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:08.499833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:08.499933 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:08.500056 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:08.501394 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.502719 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.504143 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:08.504859 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:08.505021 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:08.510059 (+ 5038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:08.510080 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:08.510102 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:08.510155 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:08.514422 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:08.515145 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:08.515155 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:08.515159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:08.515367 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:08.515751 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:08.515756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:08.515759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:08.516250 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:08.516600 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:08.516770 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 22:46:08.517037 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 22:46:08.518368 19827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:08.643992 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 19636
I20260806 22:46:08.667862 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 19083
I20260806 22:46:08.669045 19178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:08.780786 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 19083
2026-08-06T22:46:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4739 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T22:46:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:08Z Disabled control of system clock
I20260806 22:46:08.887079 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:38569
--webserver_interface=127.23.100.190
--webserver_port=0
--builtin_ntp_servers=127.23.100.148:36867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:38569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 22:46:09.238118 19916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:09.238420 19916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:09.238502 19916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:09.246344 19916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:09.246469 19916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:09.246507 19916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:09.246559 19916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:09.246604 19916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:09.255967 19916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:38569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:38569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:09.257334 19916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:09.259212 19916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:09.271154 19926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:09.271198 19928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:09.271234 19924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:09.272289 19916 server_base.cc:1034] running on GCE node
I20260806 22:46:09.273137 19916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:09.274667 19916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:09.275893 19916 hybrid_clock.cc:648] HybridClock initialized: now 1786056369275822 us; error 62 us; skew 500 ppm
I20260806 22:46:09.276280 19916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:09.278828 19916 webserver.cc:460] Webserver started at http://127.23.100.190:39371/ using document root <none> and password file <none>
I20260806 22:46:09.279561 19916 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:09.279682 19916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:09.280036 19916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:09.283099 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "8358e6502fe44cd791b554e71a1bb75d"
format_stamp: "Formatted at 2026-08-06 22:46:09 on dist-test-slave-zr1t"
server_key: "31643d25430f610cda116810404c1939"
server_key_iv: "ebeaec5f0dc7392c58cd197e988de2d7"
server_key_version: "encryptionkey@0"
I20260806 22:46:09.283986 19916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "8358e6502fe44cd791b554e71a1bb75d"
format_stamp: "Formatted at 2026-08-06 22:46:09 on dist-test-slave-zr1t"
server_key: "31643d25430f610cda116810404c1939"
server_key_iv: "ebeaec5f0dc7392c58cd197e988de2d7"
server_key_version: "encryptionkey@0"
I20260806 22:46:09.289429 19916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 22:46:09.293129 19937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:09.294588 19916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 22:46:09.294793 19916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "8358e6502fe44cd791b554e71a1bb75d"
format_stamp: "Formatted at 2026-08-06 22:46:09 on dist-test-slave-zr1t"
server_key: "31643d25430f610cda116810404c1939"
server_key_iv: "ebeaec5f0dc7392c58cd197e988de2d7"
server_key_version: "encryptionkey@0"
I20260806 22:46:09.295043 19916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:09.322211 19916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:09.323218 19916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:09.323549 19916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:09.344009 19916 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:38569
I20260806 22:46:09.344030 20010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:38569 every 8 connection(s)
I20260806 22:46:09.345425 19916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:09.349270 20011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:09.349484 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 19916
I20260806 22:46:09.349730 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:09.350220 23954 external_mini_cluster.cc:1428] Setting key 1b4e170f69254b26f03b423a6a663313
I20260806 22:46:09.363168 20011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d: Bootstrap starting.
I20260806 22:46:09.364820 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:09.352163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.23.100.190:38569)
0806 22:46:09.353248 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:09.353259 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:09.353301 (+ 42us) server_negotiation.cc:407] Connection header received
0806 22:46:09.355436 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:09.355458 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:09.355931 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:09.356258 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:09.356971 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:09.358355 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:09.360349 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:09.361079 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:09.361216 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:09.362539 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:09.362546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:09.362559 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:09.363080 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:09.363168 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:09.363191 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:09.363308 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:09.363526 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 22:46:09.363880 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":438,"threads_started":1}
I20260806 22:46:09.366539 20011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:09.367615 20011 log.cc:824] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:09.370276 20011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d: No bootstrap required, opened a new log
I20260806 22:46:09.377439 20011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8358e6502fe44cd791b554e71a1bb75d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 38569 } }
I20260806 22:46:09.377878 20011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:09.377985 20011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8358e6502fe44cd791b554e71a1bb75d, State: Initialized, Role: FOLLOWER
I20260806 22:46:09.378577 20011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8358e6502fe44cd791b554e71a1bb75d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 38569 } }
I20260806 22:46:09.378844 20011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:09.378981 20011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:09.379154 20011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:09.381390 20011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8358e6502fe44cd791b554e71a1bb75d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 38569 } }
I20260806 22:46:09.381933 20011 leader_election.cc:302] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8358e6502fe44cd791b554e71a1bb75d; no voters:
I20260806 22:46:09.382552 20011 leader_election.cc:288] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:09.382721 20020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:09.383530 20020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [term 1 LEADER]: Becoming Leader. State: Replica: 8358e6502fe44cd791b554e71a1bb75d, State: Running, Role: LEADER
I20260806 22:46:09.384023 20020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8358e6502fe44cd791b554e71a1bb75d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 38569 } }
I20260806 22:46:09.384681 20011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:09.387214 20024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8358e6502fe44cd791b554e71a1bb75d. Latest consensus state: current_term: 1 leader_uuid: "8358e6502fe44cd791b554e71a1bb75d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8358e6502fe44cd791b554e71a1bb75d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 38569 } } }
I20260806 22:46:09.387821 20024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:09.387238 20021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8358e6502fe44cd791b554e71a1bb75d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8358e6502fe44cd791b554e71a1bb75d" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 38569 } } }
I20260806 22:46:09.388252 20021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:09.390189 20034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:09.395370 20034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:09.405645 20034 catalog_manager.cc:1347] Generated new cluster ID: 45767cdfe56749b9a3eef57e843f043e
I20260806 22:46:09.405788 20034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:09.419559 20034 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:09.420832 20034 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:09.433188 20034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d: Generated new TSK 0
I20260806 22:46:09.434324 20034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:09.447319 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:38569
--builtin_ntp_servers=127.23.100.148:36867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 22:46:09.799714 20049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:09.800017 20049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:09.800158 20049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:09.807646 20049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:09.807758 20049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:09.807931 20049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:46:09.817191 20049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:38569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:09.818568 20049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:09.820377 20049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:09.833873 20059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:09.833918 20064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:09.834830 20049 server_base.cc:1034] running on GCE node
W20260806 22:46:09.834940 20060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:09.835672 20049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:09.836688 20049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:09.837927 20049 hybrid_clock.cc:648] HybridClock initialized: now 1786056369837887 us; error 28 us; skew 500 ppm
I20260806 22:46:09.838277 20049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:09.840682 20049 webserver.cc:460] Webserver started at http://127.23.100.129:45347/ using document root <none> and password file <none>
I20260806 22:46:09.841398 20049 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:09.841572 20049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:09.841912 20049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:09.844846 20049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "0a992b112e3b4235bfd74d5b73fa2289"
format_stamp: "Formatted at 2026-08-06 22:46:09 on dist-test-slave-zr1t"
server_key: "479a5136c59e81f94332acde1b4013f3"
server_key_iv: "cce4b6bf7ca1a936cdb4c04dc3c34719"
server_key_version: "encryptionkey@0"
I20260806 22:46:09.845718 20049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "0a992b112e3b4235bfd74d5b73fa2289"
format_stamp: "Formatted at 2026-08-06 22:46:09 on dist-test-slave-zr1t"
server_key: "479a5136c59e81f94332acde1b4013f3"
server_key_iv: "cce4b6bf7ca1a936cdb4c04dc3c34719"
server_key_version: "encryptionkey@0"
I20260806 22:46:09.850803 20049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 22:46:09.855472 20072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:09.856817 20049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 22:46:09.856995 20049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "0a992b112e3b4235bfd74d5b73fa2289"
format_stamp: "Formatted at 2026-08-06 22:46:09 on dist-test-slave-zr1t"
server_key: "479a5136c59e81f94332acde1b4013f3"
server_key_iv: "cce4b6bf7ca1a936cdb4c04dc3c34719"
server_key_version: "encryptionkey@0"
I20260806 22:46:09.857214 20049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:09.872905 20049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:09.873804 20049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:09.874115 20049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:09.875118 20049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:09.876612 20049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:09.876744 20049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:09.876870 20049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:09.876958 20049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:09.912405 20049 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:45439
I20260806 22:46:09.912448 20221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:45439 every 8 connection(s)
I20260806 22:46:09.913911 20049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:46:09.922866 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 20049
I20260806 22:46:09.923074 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:46:09.923520 23954 external_mini_cluster.cc:1428] Setting key 6db07b1cefb4abd3691886f4316a39d9
I20260806 22:46:09.926265 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:38569
--builtin_ntp_servers=127.23.100.148:36867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 22:46:09.926841 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:09.916533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:53865 (local address 127.23.100.190:38569)
0806 22:46:09.916976 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:09.916980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:09.918251 (+ 1271us) server_negotiation.cc:407] Connection header received
0806 22:46:09.920081 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:09.920086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:09.920181 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:09.920316 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:09.921395 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:09.922192 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:09.923945 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:09.924379 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:09.924516 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:09.925968 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:09.925974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:09.925977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:09.925995 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:09.926035 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:09.926047 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:09.926154 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:09.926378 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 22:46:09.926480 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":219,"threads_started":1}
I20260806 22:46:09.927821 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:09.917136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:38569 (local address 127.23.100.129:53865)
0806 22:46:09.918088 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:09.918132 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:09.919808 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:09.920555 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:09.920564 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:09.920649 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:09.921221 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:09.921237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:09.922358 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:09.922368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:09.923799 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:09.923812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:09.924620 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:09.924628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:09.924858 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:09.925635 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:46:09.925638 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:09.925678 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:46:09.925755 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:09.926153 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:09.926166 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:09.926177 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 22:46:09.926366 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 22:46:09.926771 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":331,"threads_started":1}
I20260806 22:46:09.929801 20223 heartbeater.cc:347] Connected to a master server at 127.23.100.190:38569
I20260806 22:46:09.930207 20223 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:09.931033 20223 heartbeater.cc:511] Master 127.23.100.190:38569 requested a full tablet report, sending...
I20260806 22:46:09.933630 19957 ts_manager.cc:194] Registered new tserver with Master: 0a992b112e3b4235bfd74d5b73fa2289 (127.23.100.129:45439)
I20260806 22:46:09.935791 19957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.129:53865
W20260806 22:46:10.274801 20231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:10.275089 20231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:10.275240 20231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:10.282645 20231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:10.282755 20231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:10.282961 20231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:46:10.292799 20231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:38569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:10.294299 20231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:10.295993 20231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:10.309249 20237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:10.309265 20239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:10.309239 20241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:10.310117 20231 server_base.cc:1034] running on GCE node
I20260806 22:46:10.310717 20231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:10.311658 20231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:10.312847 20231 hybrid_clock.cc:648] HybridClock initialized: now 1786056370312806 us; error 32 us; skew 500 ppm
I20260806 22:46:10.313136 20231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:10.315601 20231 webserver.cc:460] Webserver started at http://127.23.100.130:35281/ using document root <none> and password file <none>
I20260806 22:46:10.316252 20231 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:10.316339 20231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:10.316620 20231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:10.319628 20231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "7b90dd3018f547529a28d699bed76494"
format_stamp: "Formatted at 2026-08-06 22:46:10 on dist-test-slave-zr1t"
server_key: "14529f2c4034946941e58550d01dec30"
server_key_iv: "5db5274799f975f25f7477bf918cd727"
server_key_version: "encryptionkey@0"
I20260806 22:46:10.320362 20231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "7b90dd3018f547529a28d699bed76494"
format_stamp: "Formatted at 2026-08-06 22:46:10 on dist-test-slave-zr1t"
server_key: "14529f2c4034946941e58550d01dec30"
server_key_iv: "5db5274799f975f25f7477bf918cd727"
server_key_version: "encryptionkey@0"
I20260806 22:46:10.325378 20231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 22:46:10.328935 20253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:10.330304 20231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:46:10.330504 20231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "7b90dd3018f547529a28d699bed76494"
format_stamp: "Formatted at 2026-08-06 22:46:10 on dist-test-slave-zr1t"
server_key: "14529f2c4034946941e58550d01dec30"
server_key_iv: "5db5274799f975f25f7477bf918cd727"
server_key_version: "encryptionkey@0"
I20260806 22:46:10.330726 20231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:10.344273 20231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:10.345054 20231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:10.345363 20231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:10.346441 20231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:10.347892 20231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:10.348024 20231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:10.348148 20231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:10.348237 20231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:10.387161 20231 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:34973
I20260806 22:46:10.387219 20419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:34973 every 8 connection(s)
I20260806 22:46:10.388662 20231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:46:10.391880 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 20231
I20260806 22:46:10.392074 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:46:10.392534 23954 external_mini_cluster.cc:1428] Setting key 3e78b5066a1ebe436bcfaf7afa37c61a
I20260806 22:46:10.395478 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:38569
--builtin_ntp_servers=127.23.100.148:36867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 22:46:10.402443 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:10.391728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:38459 (local address 127.23.100.190:38569)
0806 22:46:10.391954 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:10.391960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:10.393611 (+ 1651us) server_negotiation.cc:407] Connection header received
0806 22:46:10.395525 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:10.395531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:10.395623 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:10.395720 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:10.397076 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.397880 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.399539 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.399945 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.400087 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:10.401580 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:10.401585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:10.401589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:10.401609 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:10.401639 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:10.401652 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:10.401761 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:10.401996 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 22:46:10.402099 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 22:46:10.403450 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:10.392352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:38569 (local address 127.23.100.130:38459)
0806 22:46:10.393440 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:10.393485 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:10.395232 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:10.395987 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:10.395996 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:10.396138 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:10.396816 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:10.396834 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.398055 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.398060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:10.399373 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:10.399387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.400191 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.400199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:10.400449 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:10.401201 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:46:10.401204 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:10.401245 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:46:10.401328 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:10.401762 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:10.401776 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:10.401786 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 22:46:10.401985 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 22:46:10.402402 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":359,"threads_started":1}
I20260806 22:46:10.404906 20420 heartbeater.cc:347] Connected to a master server at 127.23.100.190:38569
I20260806 22:46:10.405242 20420 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:10.406219 20420 heartbeater.cc:511] Master 127.23.100.190:38569 requested a full tablet report, sending...
I20260806 22:46:10.407856 19957 ts_manager.cc:194] Registered new tserver with Master: 7b90dd3018f547529a28d699bed76494 (127.23.100.130:34973)
I20260806 22:46:10.408716 19957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.130:38459
W20260806 22:46:10.743793 20424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:10.744128 20424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:10.744313 20424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:10.751801 20424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:10.751957 20424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:10.752146 20424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:46:10.761602 20424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:36867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:38569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:10.763008 20424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:10.764853 20424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:10.778453 20439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:10.778431 20434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:10.778692 20433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:10.779402 20424 server_base.cc:1034] running on GCE node
I20260806 22:46:10.780077 20424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:10.781101 20424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:10.782354 20424 hybrid_clock.cc:648] HybridClock initialized: now 1786056370782314 us; error 45 us; skew 500 ppm
I20260806 22:46:10.782709 20424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:10.785077 20424 webserver.cc:460] Webserver started at http://127.23.100.131:43273/ using document root <none> and password file <none>
I20260806 22:46:10.785918 20424 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:10.786063 20424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:10.786392 20424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:10.789378 20424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "2105273e515b4c988e5d87144e9e44b3"
format_stamp: "Formatted at 2026-08-06 22:46:10 on dist-test-slave-zr1t"
server_key: "212b7e51def2967f6350f02fa5fd60a8"
server_key_iv: "144a5f5cfaa3f3b2e5be46111064afec"
server_key_version: "encryptionkey@0"
I20260806 22:46:10.790329 20424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "2105273e515b4c988e5d87144e9e44b3"
format_stamp: "Formatted at 2026-08-06 22:46:10 on dist-test-slave-zr1t"
server_key: "212b7e51def2967f6350f02fa5fd60a8"
server_key_iv: "144a5f5cfaa3f3b2e5be46111064afec"
server_key_version: "encryptionkey@0"
I20260806 22:46:10.795487 20424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 22:46:10.799193 20449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:10.800647 20424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 22:46:10.800828 20424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "2105273e515b4c988e5d87144e9e44b3"
format_stamp: "Formatted at 2026-08-06 22:46:10 on dist-test-slave-zr1t"
server_key: "212b7e51def2967f6350f02fa5fd60a8"
server_key_iv: "144a5f5cfaa3f3b2e5be46111064afec"
server_key_version: "encryptionkey@0"
I20260806 22:46:10.801074 20424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:10.846047 20424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:10.846937 20424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:10.847263 20424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:10.848320 20424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:10.849949 20424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:10.850085 20424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:10.850216 20424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:10.850312 20424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:10.886410 20424 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:35809
I20260806 22:46:10.886503 20613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:35809 every 8 connection(s)
I20260806 22:46:10.887768 20424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:46:10.891563 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 20424
I20260806 22:46:10.891779 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:46:10.892545 23954 external_mini_cluster.cc:1428] Setting key 0b01547bf4d8bc55497ada058fd74a82
I20260806 22:46:10.903007 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:10.890713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:52275 (local address 127.23.100.190:38569)
0806 22:46:10.890947 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:10.890953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:10.894188 (+ 3235us) server_negotiation.cc:407] Connection header received
0806 22:46:10.896099 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:10.896103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:10.896170 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:10.896251 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:10.897377 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.898357 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.900222 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.900670 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.900781 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:10.902202 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:10.902207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:10.902210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:10.902229 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:10.902276 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:10.902289 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:10.902401 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:10.902667 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 22:46:10.902757 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 22:46:10.903975 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:10.891410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:38569 (local address 127.23.100.131:52275)
0806 22:46:10.894015 (+ 2605us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:10.894064 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:10.895834 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:10.896478 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:10.896486 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:10.896577 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:10.897184 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:10.897203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.898548 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.898553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:10.900030 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:10.900044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.900877 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.900882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:10.901084 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:10.901858 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:46:10.901860 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:10.901909 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:46:10.901994 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:10.902412 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:10.902420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:10.902426 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 22:46:10.902591 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 22:46:10.902890 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1996,"thread_start_us":1774,"threads_started":1}
I20260806 22:46:10.905619 20614 heartbeater.cc:347] Connected to a master server at 127.23.100.190:38569
I20260806 22:46:10.906036 20614 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:10.907040 20614 heartbeater.cc:511] Master 127.23.100.190:38569 requested a full tablet report, sending...
I20260806 22:46:10.908766 19957 ts_manager.cc:194] Registered new tserver with Master: 2105273e515b4c988e5d87144e9e44b3 (127.23.100.131:35809)
I20260806 22:46:10.909749 19957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.131:52275
I20260806 22:46:10.921161 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:46:10.932767 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:10.925891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.23.100.190:38569)
0806 22:46:10.926077 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:10.926082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:10.926540 (+ 458us) server_negotiation.cc:407] Connection header received
0806 22:46:10.926696 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:10.926701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:10.926766 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:10.926855 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:10.927389 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.928093 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.929569 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.930013 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.930114 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:10.931451 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:10.931460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:10.931467 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:10.931494 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:10.931552 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:10.931573 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:10.931749 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:10.932273 (+ 524us) server_negotiation.cc:299] Negotiation successful
0806 22:46:10.932337 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 22:46:10.939440 20223 heartbeater.cc:503] Master 127.23.100.190:38569 was elected leader, sending a full tablet report...
I20260806 22:46:10.936914 19957 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:46:10.944506 19957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:46:10.972869 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:10.957217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:35809 (local address 127.0.0.1:55606)
0806 22:46:10.958513 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:10.958562 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:10.958794 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:10.961656 (+ 2862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:10.961666 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:10.961705 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:10.962309 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:10.962327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.965180 (+ 2853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.965189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:10.968753 (+ 3564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:10.968780 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.971670 (+ 2890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.971685 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:10.972031 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:10.972118 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 22:46:10.972242 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 22:46:10.972459 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":1,"thread_start_us":1821,"threads_started":1}
I20260806 22:46:10.974336 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:10.957614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.23.100.131:35809)
0806 22:46:10.960954 (+ 3340us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:10.960962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:10.960990 (+ 28us) server_negotiation.cc:407] Connection header received
0806 22:46:10.961070 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:10.961076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:10.961269 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:10.961570 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:10.962710 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.964953 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.969169 (+ 4216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.971396 (+ 2227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.972894 (+ 1498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:10.973687 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:10.973780 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 22:46:10.973948 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3189,"thread_start_us":3029,"threads_started":1}
I20260806 22:46:10.975364 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:10.957545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45439 (local address 127.0.0.1:45380)
0806 22:46:10.961897 (+ 4352us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:10.961920 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:10.962747 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:10.964032 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:10.964039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:10.964076 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:10.964605 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:10.964621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.968837 (+ 4216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.968851 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:10.972529 (+ 3678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:10.972544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.974529 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.974536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:10.974730 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:10.974767 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 22:46:10.974858 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 22:46:10.975007 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4194,"mutex_wait_us":65,"thread_start_us":4013,"threads_started":1}
I20260806 22:46:10.976565 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:10.958564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45380 (local address 127.23.100.129:45439)
0806 22:46:10.960434 (+ 1870us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:10.960447 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:10.962028 (+ 1581us) server_negotiation.cc:407] Connection header received
0806 22:46:10.962999 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:10.963011 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:10.963311 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:10.963644 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:10.964832 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.968593 (+ 3761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.972754 (+ 4161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.974293 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.974520 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:10.975945 (+ 1425us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:10.976020 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 22:46:10.976183 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1578,"thread_start_us":1399,"threads_started":1}
I20260806 22:46:10.978816 20519 tablet_service.cc:1467] Processing CreateTablet for tablet 67f53aa7a4654dc9aeb33ccff581d8e4 (DEFAULT_TABLE table=test-table [id=adc9eeb225bd40e2ab1860bec3c60311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:10.980460 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:10.962302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:34973 (local address 127.0.0.1:47136)
0806 22:46:10.972031 (+ 9729us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:10.972053 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:10.972200 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:10.973346 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:10.973350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:10.973371 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:10.973782 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:10.973792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.975730 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.975736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:10.977703 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:10.977718 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.979599 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.979607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:10.979805 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:10.979842 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 22:46:10.979974 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 22:46:10.980126 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9574,"thread_start_us":9986,"threads_started":1}
I20260806 22:46:10.980474 20519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f53aa7a4654dc9aeb33ccff581d8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:10.980126 20138 tablet_service.cc:1467] Processing CreateTablet for tablet 67f53aa7a4654dc9aeb33ccff581d8e4 (DEFAULT_TABLE table=test-table [id=adc9eeb225bd40e2ab1860bec3c60311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:10.980764 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:10.962332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.23.100.130:34973)
0806 22:46:10.971474 (+ 9142us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:10.971487 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:10.972485 (+ 998us) server_negotiation.cc:407] Connection header received
0806 22:46:10.972614 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:10.972623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:10.972902 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:10.973131 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:10.973973 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.975526 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.977943 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:10.979360 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:10.979622 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:10.979812 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:10.980044 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 22:46:10.980210 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8948,"thread_start_us":8804,"threads_started":1}
I20260806 22:46:10.981755 20138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f53aa7a4654dc9aeb33ccff581d8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:10.984577 20326 tablet_service.cc:1467] Processing CreateTablet for tablet 67f53aa7a4654dc9aeb33ccff581d8e4 (DEFAULT_TABLE table=test-table [id=adc9eeb225bd40e2ab1860bec3c60311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:10.985981 20326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f53aa7a4654dc9aeb33ccff581d8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:10.995582 20642 tablet_bootstrap.cc:492] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289: Bootstrap starting.
I20260806 22:46:10.997997 20644 tablet_bootstrap.cc:492] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: Bootstrap starting.
I20260806 22:46:11.000451 20642 tablet_bootstrap.cc:654] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:11.001962 20642 log.cc:824] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:11.002099 20644 tablet_bootstrap.cc:654] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:11.002622 20645 tablet_bootstrap.cc:492] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: Bootstrap starting.
I20260806 22:46:11.003734 20644 log.cc:824] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:11.006002 20645 tablet_bootstrap.cc:654] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:11.006939 20645 log.cc:824] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:11.009734 20645 tablet_bootstrap.cc:492] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: No bootstrap required, opened a new log
I20260806 22:46:11.010311 20645 ts_tablet_manager.cc:1397] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260806 22:46:11.010550 20642 tablet_bootstrap.cc:492] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289: No bootstrap required, opened a new log
I20260806 22:46:11.010850 20642 ts_tablet_manager.cc:1397] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260806 22:46:11.012825 20644 tablet_bootstrap.cc:492] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: No bootstrap required, opened a new log
I20260806 22:46:11.013123 20644 ts_tablet_manager.cc:1397] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260806 22:46:11.017773 20645 raft_consensus.cc:348] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2105273e515b4c988e5d87144e9e44b3" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35809 } } peers { permanent_uuid: "0a992b112e3b4235bfd74d5b73fa2289" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45439 } } peers { permanent_uuid: "7b90dd3018f547529a28d699bed76494" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 34973 } }
I20260806 22:46:11.018182 20645 raft_consensus.cc:374] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:11.018282 20645 raft_consensus.cc:729] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b90dd3018f547529a28d699bed76494, State: Initialized, Role: FOLLOWER
I20260806 22:46:11.018921 20645 consensus_queue.cc:260] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2105273e515b4c988e5d87144e9e44b3" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35809 } } peers { permanent_uuid: "0a992b112e3b4235bfd74d5b73fa2289" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45439 } } peers { permanent_uuid: "7b90dd3018f547529a28d699bed76494" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 34973 } }
I20260806 22:46:11.020068 20420 heartbeater.cc:503] Master 127.23.100.190:38569 was elected leader, sending a full tablet report...
I20260806 22:46:11.021900 20645 ts_tablet_manager.cc:1428] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260806 22:46:11.022686 20642 raft_consensus.cc:348] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2105273e515b4c988e5d87144e9e44b3" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35809 } } peers { permanent_uuid: "0a992b112e3b4235bfd74d5b73fa2289" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45439 } } peers { permanent_uuid: "7b90dd3018f547529a28d699bed76494" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 34973 } }
I20260806 22:46:11.023239 20642 raft_consensus.cc:374] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:11.023357 20642 raft_consensus.cc:729] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a992b112e3b4235bfd74d5b73fa2289, State: Initialized, Role: FOLLOWER
I20260806 22:46:11.024236 20642 consensus_queue.cc:260] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2105273e515b4c988e5d87144e9e44b3" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35809 } } peers { permanent_uuid: "0a992b112e3b4235bfd74d5b73fa2289" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45439 } } peers { permanent_uuid: "7b90dd3018f547529a28d699bed76494" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 34973 } }
I20260806 22:46:11.026376 20642 ts_tablet_manager.cc:1428] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260806 22:46:11.026539 20644 raft_consensus.cc:348] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2105273e515b4c988e5d87144e9e44b3" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35809 } } peers { permanent_uuid: "0a992b112e3b4235bfd74d5b73fa2289" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45439 } } peers { permanent_uuid: "7b90dd3018f547529a28d699bed76494" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 34973 } }
I20260806 22:46:11.027014 20644 raft_consensus.cc:374] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:11.027125 20644 raft_consensus.cc:729] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2105273e515b4c988e5d87144e9e44b3, State: Initialized, Role: FOLLOWER
I20260806 22:46:11.027681 20644 consensus_queue.cc:260] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2105273e515b4c988e5d87144e9e44b3" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35809 } } peers { permanent_uuid: "0a992b112e3b4235bfd74d5b73fa2289" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45439 } } peers { permanent_uuid: "7b90dd3018f547529a28d699bed76494" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 34973 } }
I20260806 22:46:11.028827 20614 heartbeater.cc:503] Master 127.23.100.190:38569 was elected leader, sending a full tablet report...
I20260806 22:46:11.029330 20644 ts_tablet_manager.cc:1428] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
W20260806 22:46:11.139729 20617 tablet.cc:2369] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:46:11.140941 20422 tablet.cc:2369] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:46:11.166425 20226 tablet.cc:2369] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:46:11.273859 20656 raft_consensus.cc:482] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:46:11.274308 20656 raft_consensus.cc:504] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2105273e515b4c988e5d87144e9e44b3" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35809 } } peers { permanent_uuid: "0a992b112e3b4235bfd74d5b73fa2289" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45439 } } peers { permanent_uuid: "7b90dd3018f547529a28d699bed76494" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 34973 } }
I20260806 22:46:11.276127 20656 leader_election.cc:288] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a992b112e3b4235bfd74d5b73fa2289 (127.23.100.129:45439), 7b90dd3018f547529a28d699bed76494 (127.23.100.130:34973)
I20260806 22:46:11.283351 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.276391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45439 (local address 127.23.100.131:52495)
0806 22:46:11.276599 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:11.276619 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:11.276774 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:11.277501 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:11.277506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:11.277558 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:11.277765 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:11.277780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.279244 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.279250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:11.280881 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:11.280894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.282526 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.282532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:11.282741 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:11.282777 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 22:46:11.282881 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 22:46:11.283024 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":29}
I20260806 22:46:11.283324 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.276717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:52495 (local address 127.23.100.129:45439)
0806 22:46:11.276939 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:11.276946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:11.276970 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:46:11.277086 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:11.277095 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:11.277214 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:11.277341 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:11.277990 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.279109 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.281070 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.282339 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.282478 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:11.282532 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:11.282918 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 22:46:11.283027 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 22:46:11.284159 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.277318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:44265 (local address 127.23.100.130:34973)
0806 22:46:11.277550 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:11.277557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:11.277754 (+ 197us) server_negotiation.cc:407] Connection header received
0806 22:46:11.277926 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:11.277934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:11.278036 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:11.278105 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:11.278779 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.279649 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.281987 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.283015 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.283161 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:11.283231 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:11.283663 (+ 432us) server_negotiation.cc:299] Negotiation successful
0806 22:46:11.283855 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 22:46:11.284706 20160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f53aa7a4654dc9aeb33ccff581d8e4" candidate_uuid: "2105273e515b4c988e5d87144e9e44b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a992b112e3b4235bfd74d5b73fa2289" is_pre_election: true
I20260806 22:46:11.284896 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.276856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:34973 (local address 127.23.100.131:44265)
0806 22:46:11.277646 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:11.277663 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:11.277777 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:11.278168 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:11.278173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:11.278203 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:11.278609 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:11.278622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.279819 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.279823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:11.281760 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:11.281777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.283202 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.283209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:11.283433 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:11.283473 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 22:46:11.284375 (+ 902us) client_negotiation.cc:240] Negotiation successful
0806 22:46:11.284537 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":4,"thread_start_us":381,"threads_started":1}
I20260806 22:46:11.285270 20160 raft_consensus.cc:2454] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2105273e515b4c988e5d87144e9e44b3 in term 0.
I20260806 22:46:11.285991 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f53aa7a4654dc9aeb33ccff581d8e4" candidate_uuid: "2105273e515b4c988e5d87144e9e44b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b90dd3018f547529a28d699bed76494" is_pre_election: true
I20260806 22:46:11.286103 20451 leader_election.cc:302] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a992b112e3b4235bfd74d5b73fa2289, 2105273e515b4c988e5d87144e9e44b3; no voters:
I20260806 22:46:11.286558 20656 raft_consensus.cc:2781] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:46:11.286551 20349 raft_consensus.cc:2454] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2105273e515b4c988e5d87144e9e44b3 in term 0.
I20260806 22:46:11.286734 20656 raft_consensus.cc:482] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:46:11.286854 20656 raft_consensus.cc:3037] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:11.289106 20656 raft_consensus.cc:504] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2105273e515b4c988e5d87144e9e44b3" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35809 } } peers { permanent_uuid: "0a992b112e3b4235bfd74d5b73fa2289" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45439 } } peers { permanent_uuid: "7b90dd3018f547529a28d699bed76494" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 34973 } }
I20260806 22:46:11.289778 20656 leader_election.cc:288] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [CANDIDATE]: Term 1 election: Requested vote from peers 0a992b112e3b4235bfd74d5b73fa2289 (127.23.100.129:45439), 7b90dd3018f547529a28d699bed76494 (127.23.100.130:34973)
I20260806 22:46:11.290340 20160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f53aa7a4654dc9aeb33ccff581d8e4" candidate_uuid: "2105273e515b4c988e5d87144e9e44b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a992b112e3b4235bfd74d5b73fa2289"
I20260806 22:46:11.290442 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f53aa7a4654dc9aeb33ccff581d8e4" candidate_uuid: "2105273e515b4c988e5d87144e9e44b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b90dd3018f547529a28d699bed76494"
I20260806 22:46:11.290601 20160 raft_consensus.cc:3037] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:11.290684 20349 raft_consensus.cc:3037] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:11.292882 20160 raft_consensus.cc:2454] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2105273e515b4c988e5d87144e9e44b3 in term 1.
I20260806 22:46:11.293309 20349 raft_consensus.cc:2454] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2105273e515b4c988e5d87144e9e44b3 in term 1.
I20260806 22:46:11.293339 20451 leader_election.cc:302] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a992b112e3b4235bfd74d5b73fa2289, 2105273e515b4c988e5d87144e9e44b3; no voters:
I20260806 22:46:11.293696 20656 raft_consensus.cc:2781] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:11.294190 20656 raft_consensus.cc:686] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 1 LEADER]: Becoming Leader. State: Replica: 2105273e515b4c988e5d87144e9e44b3, State: Running, Role: LEADER
I20260806 22:46:11.294682 20656 consensus_queue.cc:237] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2105273e515b4c988e5d87144e9e44b3" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35809 } } peers { permanent_uuid: "0a992b112e3b4235bfd74d5b73fa2289" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45439 } } peers { permanent_uuid: "7b90dd3018f547529a28d699bed76494" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 34973 } }
I20260806 22:46:11.299834 19956 catalog_manager.cc:5495] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2105273e515b4c988e5d87144e9e44b3 (127.23.100.131). New cstate: current_term: 1 leader_uuid: "2105273e515b4c988e5d87144e9e44b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2105273e515b4c988e5d87144e9e44b3" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 35809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a992b112e3b4235bfd74d5b73fa2289" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b90dd3018f547529a28d699bed76494" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 34973 } health_report { overall_health: UNKNOWN } } }
I20260806 22:46:11.341182 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.336059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.23.100.131:35809)
0806 22:46:11.336229 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:11.336234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:11.336253 (+ 19us) server_negotiation.cc:407] Connection header received
0806 22:46:11.336378 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:11.336381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:11.336464 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:11.336593 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:46:11.336966 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.337730 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.339106 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.339510 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.339599 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:11.340109 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:46:11.340634 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:46:11.340762 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:11.340932 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 22:46:11.340982 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 22:46:11.350294 20160 raft_consensus.cc:1261] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289 [term 1 FOLLOWER]: Refusing update from remote peer 2105273e515b4c988e5d87144e9e44b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:11.350294 20349 raft_consensus.cc:1261] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494 [term 1 FOLLOWER]: Refusing update from remote peer 2105273e515b4c988e5d87144e9e44b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:11.351397 20658 consensus_queue.cc:1035] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b90dd3018f547529a28d699bed76494" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:46:11.351861 20656 consensus_queue.cc:1035] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a992b112e3b4235bfd74d5b73fa2289" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:46:11.364478 20663 mvcc.cc:205] Tried to move back new op lower bound from 7315686897039101952 to 7315686896827056128. Current Snapshot: MvccSnapshot[applied={T|T < 7315686897039101952}]
I20260806 22:46:11.366318 20664 mvcc.cc:205] Tried to move back new op lower bound from 7315686897039101952 to 7315686896827056128. Current Snapshot: MvccSnapshot[applied={T|T < 7315686897039101952}]
I20260806 22:46:11.382048 19956 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:11.382364 19956 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:11.386420 19956 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8358e6502fe44cd791b554e71a1bb75d: Sending DeleteTablet for 3 replicas of tablet 67f53aa7a4654dc9aeb33ccff581d8e4
I20260806 22:46:11.387666 20519 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f53aa7a4654dc9aeb33ccff581d8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:11 UTC) from {username='slave'} at 127.0.0.1:55606
I20260806 22:46:11.388288 20326 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f53aa7a4654dc9aeb33ccff581d8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:11 UTC) from {username='slave'} at 127.0.0.1:47136
I20260806 22:46:11.390053 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 20049
I20260806 22:46:11.391484 20138 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f53aa7a4654dc9aeb33ccff581d8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:11 UTC) from {username='slave'} at 127.0.0.1:45380
I20260806 22:46:11.422050 20674 tablet_replica.cc:330] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: stopping tablet replica
I20260806 22:46:11.422737 20678 tablet_replica.cc:330] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289: stopping tablet replica
I20260806 22:46:11.423070 20676 tablet_replica.cc:330] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: stopping tablet replica
I20260806 22:46:11.423235 20678 raft_consensus.cc:2229] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:11.423651 20676 raft_consensus.cc:2229] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:11.423796 20678 raft_consensus.cc:2258] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:11.424286 20676 raft_consensus.cc:2258] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:11.424530 20674 raft_consensus.cc:2229] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:46:11.425264 20674 raft_consensus.cc:2258] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:11.427014 20676 ts_tablet_manager.cc:1905] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:11.428915 20674 ts_tablet_manager.cc:1905] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:11.434118 20678 ts_tablet_manager.cc:1905] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 0a992b112e3b4235bfd74d5b73fa2289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:11.434417 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.391517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.23.100.129:45439)
0806 22:46:11.391702 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:11.391708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:11.416286 (+ 24578us) server_negotiation.cc:407] Connection header received
0806 22:46:11.416923 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:11.416945 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:11.417108 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:11.417202 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:11.418176 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.419453 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.425639 (+ 6186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.426287 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.426400 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:11.432208 (+ 5808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:11.432218 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:11.432236 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:11.433070 (+ 834us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:11.433139 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:11.433167 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:11.433279 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:11.433875 (+ 596us) server_negotiation.cc:299] Negotiation successful
0806 22:46:11.434006 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 22:46:11.435691 20216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:11.435673 20676 ts_tablet_manager.cc:1918] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:11.435949 20676 log.cc:1196] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/67f53aa7a4654dc9aeb33ccff581d8e4
I20260806 22:46:11.436748 20676 ts_tablet_manager.cc:1939] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 7b90dd3018f547529a28d699bed76494: Deleting consensus metadata
I20260806 22:46:11.439167 19939 catalog_manager.cc:4841] TS 7b90dd3018f547529a28d699bed76494 (127.23.100.130:34973): tablet 67f53aa7a4654dc9aeb33ccff581d8e4 (table test-table [id=adc9eeb225bd40e2ab1860bec3c60311]) successfully deleted
I20260806 22:46:11.441074 20674 ts_tablet_manager.cc:1918] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:11.441256 20674 log.cc:1196] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/67f53aa7a4654dc9aeb33ccff581d8e4
I20260806 22:46:11.441905 20674 ts_tablet_manager.cc:1939] T 67f53aa7a4654dc9aeb33ccff581d8e4 P 2105273e515b4c988e5d87144e9e44b3: Deleting consensus metadata
I20260806 22:46:11.444643 19938 catalog_manager.cc:4841] TS 2105273e515b4c988e5d87144e9e44b3 (127.23.100.131:35809): tablet 67f53aa7a4654dc9aeb33ccff581d8e4 (table test-table [id=adc9eeb225bd40e2ab1860bec3c60311]) successfully deleted
I20260806 22:46:11.572317 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 20049
I20260806 22:46:11.604032 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 20231
W20260806 22:46:11.604031 19939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.100.129:45439 (error 108)
I20260806 22:46:11.606803 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.605793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45439 (local address 127.0.0.1:45394)
0806 22:46:11.606537 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:11.606643 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":519,"threads_started":1}
W20260806 22:46:11.607095 19939 catalog_manager.cc:4584] TS 0a992b112e3b4235bfd74d5b73fa2289 (127.23.100.129:45439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67f53aa7a4654dc9aeb33ccff581d8e4: Network error: Client connection negotiation failed: client connection to 127.23.100.129:45439: connect: Connection refused (error 111)
I20260806 22:46:11.612497 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.605352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.23.100.130:34973)
0806 22:46:11.605609 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:11.605635 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:11.605657 (+ 22us) server_negotiation.cc:407] Connection header received
0806 22:46:11.605726 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:11.605731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:11.605832 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:11.605983 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:11.606485 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.607619 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.608924 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.609626 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.609762 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:11.610825 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:11.610837 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:11.610855 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:11.611675 (+ 820us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:11.611737 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:11.611762 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:11.611884 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:11.611995 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 22:46:11.612108 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 22:46:11.613302 20415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:11.660869 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.660263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45439 (local address 127.0.0.1:45396)
0806 22:46:11.660536 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:11.660654 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 22:46:11.662173 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.661726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45439 (local address 127.0.0.1:45410)
0806 22:46:11.661893 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:11.661991 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 22:46:11.704156 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 20231
I20260806 22:46:11.722918 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.722467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45439 (local address 127.0.0.1:45426)
0806 22:46:11.722674 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:11.722761 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 22:46:11.724066 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.723698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45439 (local address 127.0.0.1:45428)
0806 22:46:11.723846 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:11.723924 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 22:46:11.737227 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 20424
I20260806 22:46:11.745096 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.738304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.23.100.131:35809)
0806 22:46:11.738522 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:11.738528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:11.738549 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:46:11.738621 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:11.738634 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:11.738718 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:11.738848 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:11.739285 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.740104 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.741405 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:11.742111 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:11.742273 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:11.743579 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:11.743590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:11.743607 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:11.744250 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:11.744313 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:11.744341 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:11.744468 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:11.744625 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 22:46:11.744735 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 22:46:11.746059 20608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:11.804703 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.804141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45439 (local address 127.0.0.1:45434)
0806 22:46:11.804423 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:11.804537 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260806 22:46:11.805935 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:11.805445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45439 (local address 127.0.0.1:45448)
0806 22:46:11.805704 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:11.805778 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260806 22:46:11.872525 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 20424
I20260806 22:46:11.898777 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 19916
I20260806 22:46:11.900002 20006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:12.010109 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:12.009475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45439 (local address 127.0.0.1:45450)
0806 22:46:12.009759 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:12.009891 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.129:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 22:46:12.010272 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 19916
2026-08-06T22:46:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3228 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:12 dist-test-slave-zr1t krb5kdc[20693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:12 dist-test-slave-zr1t krb5kdc[20693](info): set up 2 sockets
Aug 06 22:46:12 dist-test-slave-zr1t krb5kdc[20693](info): commencing operation
krb5kdc: starting...
W20260806 22:46:13.412626 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.348s user 0.004s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:13 dist-test-slave-zr1t krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:46:13.773408 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:43289
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:43289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:14.130087 20709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:14.130414 20709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:14.130589 20709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:14.138201 20709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:14.138334 20709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:14.138451 20709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:14.138564 20709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:14.138644 20709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:14.148932 20709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:43289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:43289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:14.150415 20709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:14.151759 20709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:14.164791 20717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:14.164731 20716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:14.165012 20719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:14.166128 20709 server_base.cc:1034] running on GCE node
I20260806 22:46:14.167001 20709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:14.168533 20709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:14.169754 20709 hybrid_clock.cc:648] HybridClock initialized: now 1786056374169700 us; error 45 us; skew 500 ppm
I20260806 22:46:14.170147 20709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:14 dist-test-slave-zr1t krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056374, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:14.173715 20709 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:46:14.175485 20709 webserver.cc:460] Webserver started at http://127.23.100.190:35815/ using document root <none> and password file <none>
I20260806 22:46:14.176242 20709 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:14.176409 20709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:14.176750 20709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:14.180094 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "edeb600022f8423dbf48b3f387bbded8"
format_stamp: "Formatted at 2026-08-06 22:46:14 on dist-test-slave-zr1t"
server_key: "b9e1b7d0551543ee60bfe15f4c3933f6"
server_key_iv: "0827714873eb2a80e639bd0863bcb9b0"
server_key_version: "encryptionkey@0"
I20260806 22:46:14.180982 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "edeb600022f8423dbf48b3f387bbded8"
format_stamp: "Formatted at 2026-08-06 22:46:14 on dist-test-slave-zr1t"
server_key: "b9e1b7d0551543ee60bfe15f4c3933f6"
server_key_iv: "0827714873eb2a80e639bd0863bcb9b0"
server_key_version: "encryptionkey@0"
I20260806 22:46:14.186873 20709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 22:46:14.190747 20727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:14.192320 20709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:46:14.192572 20709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "edeb600022f8423dbf48b3f387bbded8"
format_stamp: "Formatted at 2026-08-06 22:46:14 on dist-test-slave-zr1t"
server_key: "b9e1b7d0551543ee60bfe15f4c3933f6"
server_key_iv: "0827714873eb2a80e639bd0863bcb9b0"
server_key_version: "encryptionkey@0"
I20260806 22:46:14.192833 20709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:14.212666 20709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:14.217063 20709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:14.217393 20709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:14.237421 20709 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:43289
I20260806 22:46:14.237458 20803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:43289 every 8 connection(s)
I20260806 22:46:14.238900 20709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:14.239562 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 20709
I20260806 22:46:14.239775 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:14.240414 23954 external_mini_cluster.cc:1428] Setting key 93cb9dfa7f3f69c44a95cb75661319dc
I20260806 22:46:14.243542 20804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 22:46:14 dist-test-slave-zr1t krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:46:14.254437 20804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8: Bootstrap starting.
I20260806 22:46:14.258033 20804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:14.259115 20804 log.cc:824] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:14.262415 20804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8: No bootstrap required, opened a new log
I20260806 22:46:14.262985 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:14.242365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.23.100.190:43289)
0806 22:46:14.243167 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:14.243179 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:14.243218 (+ 39us) server_negotiation.cc:407] Connection header received
0806 22:46:14.244960 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:14.244985 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:14.245343 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:14.245764 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:14.247301 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:14.248657 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:14.250038 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:14.250708 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:14.250911 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:14.255321 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:14.255330 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:14.255344 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:14.255381 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:14.258659 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:14.259407 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:14.259414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:14.259420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:14.259671 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:14.260155 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:14.260165 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:14.260169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:14.260764 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:14.260991 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:14.261445 (+ 454us) server_negotiation.cc:299] Negotiation successful
0806 22:46:14.261860 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":357,"threads_started":1}
I20260806 22:46:14.270673 20804 raft_consensus.cc:348] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edeb600022f8423dbf48b3f387bbded8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43289 } }
I20260806 22:46:14.271128 20804 raft_consensus.cc:374] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:14.271245 20804 raft_consensus.cc:729] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edeb600022f8423dbf48b3f387bbded8, State: Initialized, Role: FOLLOWER
I20260806 22:46:14.271787 20804 consensus_queue.cc:260] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edeb600022f8423dbf48b3f387bbded8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43289 } }
I20260806 22:46:14.272006 20804 raft_consensus.cc:388] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:14.272105 20804 raft_consensus.cc:482] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:14.272234 20804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:14.274564 20804 raft_consensus.cc:504] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edeb600022f8423dbf48b3f387bbded8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43289 } }
I20260806 22:46:14.275120 20804 leader_election.cc:302] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edeb600022f8423dbf48b3f387bbded8; no voters:
I20260806 22:46:14.275707 20804 leader_election.cc:288] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:14.275842 20810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:14.276567 20810 raft_consensus.cc:686] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [term 1 LEADER]: Becoming Leader. State: Replica: edeb600022f8423dbf48b3f387bbded8, State: Running, Role: LEADER
I20260806 22:46:14.277102 20810 consensus_queue.cc:237] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edeb600022f8423dbf48b3f387bbded8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43289 } }
I20260806 22:46:14.277861 20804 sys_catalog.cc:564] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:14.281073 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edeb600022f8423dbf48b3f387bbded8. Latest consensus state: current_term: 1 leader_uuid: "edeb600022f8423dbf48b3f387bbded8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edeb600022f8423dbf48b3f387bbded8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43289 } } }
I20260806 22:46:14.281203 20811 sys_catalog.cc:455] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edeb600022f8423dbf48b3f387bbded8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edeb600022f8423dbf48b3f387bbded8" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43289 } } }
I20260806 22:46:14.281508 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:14.281535 20811 sys_catalog.cc:458] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:14.283192 20818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:14.290463 20818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:14.300520 20818 catalog_manager.cc:1347] Generated new cluster ID: 0aee22829c244faebde5175e4f515015
I20260806 22:46:14.300653 20818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:14.317054 20818 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:14.319016 20818 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:14.329282 20818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8: Generated new TSK 0
I20260806 22:46:14.330425 20818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.100.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
Entry for principal kudu/127.23.100.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab.
I20260806 22:46:14.404733 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43289
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:14.759711 20839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:14.759979 20839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:14.760154 20839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:14.767647 20839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:14.767764 20839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:14.767983 20839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:46:14.777968 20839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.129.keytab
--principal=kudu/127.23.100.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:14.779346 20839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:14.780704 20839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:14.794891 20848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:14.794948 20844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:14.794974 20845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:14.795773 20839 server_base.cc:1034] running on GCE node
I20260806 22:46:14.796375 20839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:14.797335 20839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:14.798544 20839 hybrid_clock.cc:648] HybridClock initialized: now 1786056374798437 us; error 119 us; skew 500 ppm
I20260806 22:46:14.798880 20839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:14 dist-test-slave-zr1t krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056374, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:14.802345 20839 init.cc:377] Logged in from keytab as kudu/127.23.100.129@KRBTEST.COM (short username kudu)
I20260806 22:46:14.803882 20839 webserver.cc:460] Webserver started at http://127.23.100.129:41265/ using document root <none> and password file <none>
I20260806 22:46:14.804589 20839 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:14.804744 20839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:14.805078 20839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:14.808076 20839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "fa00671e49854dfc8683872990595dc9"
format_stamp: "Formatted at 2026-08-06 22:46:14 on dist-test-slave-zr1t"
server_key: "6739352a0ffac8fa3dcc2df2b833fa87"
server_key_iv: "80d6c94cef1f66a4cffbb834bb19b7a3"
server_key_version: "encryptionkey@0"
I20260806 22:46:14.808862 20839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "fa00671e49854dfc8683872990595dc9"
format_stamp: "Formatted at 2026-08-06 22:46:14 on dist-test-slave-zr1t"
server_key: "6739352a0ffac8fa3dcc2df2b833fa87"
server_key_iv: "80d6c94cef1f66a4cffbb834bb19b7a3"
server_key_version: "encryptionkey@0"
I20260806 22:46:14.814237 20839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 22:46:14.817876 20862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:14.819171 20839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:46:14.819365 20839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "fa00671e49854dfc8683872990595dc9"
format_stamp: "Formatted at 2026-08-06 22:46:14 on dist-test-slave-zr1t"
server_key: "6739352a0ffac8fa3dcc2df2b833fa87"
server_key_iv: "80d6c94cef1f66a4cffbb834bb19b7a3"
server_key_version: "encryptionkey@0"
I20260806 22:46:14.819589 20839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:14.851428 20839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:14.855757 20839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:14.856089 20839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:14.857127 20839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:14.858592 20839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:14.858707 20839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:14.858873 20839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:14.858956 20839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:14.896433 20839 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:46015
I20260806 22:46:14.896531 21011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:46015 every 8 connection(s)
I20260806 22:46:14.898049 20839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:46:14.899645 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 20839
I20260806 22:46:14.899849 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:46:14.900388 23954 external_mini_cluster.cc:1428] Setting key 4d131f0025d0e2d017e607d89219d0ad
Aug 06 22:46:14 dist-test-slave-zr1t krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056374, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.129@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:46:14.919226 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:14.901053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:48387 (local address 127.23.100.190:43289)
0806 22:46:14.901588 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:14.901595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:14.902762 (+ 1167us) server_negotiation.cc:407] Connection header received
0806 22:46:14.904690 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:14.904696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:14.904783 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:14.904919 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:14.906516 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:14.907245 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:14.908824 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:14.909236 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:14.909376 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:14.912943 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:14.912956 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:14.912959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:14.912992 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:14.915555 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:14.916347 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:14.916353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:14.916357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:14.916544 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:14.917041 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:14.917047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:14.917052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:14.917487 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:14.917678 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:14.918457 (+ 779us) server_negotiation.cc:299] Negotiation successful
0806 22:46:14.918734 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":254,"threads_started":1}
I20260806 22:46:14.919858 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:14.901770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:43289 (local address 127.23.100.129:48387)
0806 22:46:14.902613 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:14.902660 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:14.904415 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:14.905167 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:14.905175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:14.905778 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:14.906333 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:14.906351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:14.907427 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:14.907431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:14.908654 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:14.908667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:14.909494 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:14.909502 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:14.909768 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:14.910502 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:46:14.910524 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:14.912717 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:14.915717 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:14.915727 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:14.915743 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:14.916203 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:14.916689 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:14.916692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:14.916695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:14.916907 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:14.917678 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:14.917685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:14.918037 (+ 352us) client_negotiation.cc:769] Sending connection context
0806 22:46:14.918276 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 22:46:14.918683 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":389,"threads_started":1}
I20260806 22:46:14.921682 21013 heartbeater.cc:347] Connected to a master server at 127.23.100.190:43289
I20260806 22:46:14.922036 21013 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:14.922941 21013 heartbeater.cc:511] Master 127.23.100.190:43289 requested a full tablet report, sending...
I20260806 22:46:14.925249 20753 ts_manager.cc:194] Registered new tserver with Master: fa00671e49854dfc8683872990595dc9 (127.23.100.129:46015)
I20260806 22:46:14.926832 20753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.129@KRBTEST.COM'} at 127.23.100.129:48387
WARNING: no policy specified for kudu/127.23.100.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
Entry for principal kudu/127.23.100.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab.
I20260806 22:46:14.979437 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43289
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:15.330082 21025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:15.330492 21025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:15.330675 21025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:15.338378 21025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:15.338491 21025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:15.338666 21025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:46:15.348671 21025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.130.keytab
--principal=kudu/127.23.100.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:15.350034 21025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:15.351326 21025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:15.364823 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:15.364903 21033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:15.364874 21035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:15.365981 21025 server_base.cc:1034] running on GCE node
I20260806 22:46:15.366634 21025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:15.367537 21025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:15.368748 21025 hybrid_clock.cc:648] HybridClock initialized: now 1786056375368711 us; error 36 us; skew 500 ppm
I20260806 22:46:15.369031 21025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:15 dist-test-slave-zr1t krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056375, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:15.372282 21025 init.cc:377] Logged in from keytab as kudu/127.23.100.130@KRBTEST.COM (short username kudu)
I20260806 22:46:15.373868 21025 webserver.cc:460] Webserver started at http://127.23.100.130:46741/ using document root <none> and password file <none>
I20260806 22:46:15.374548 21025 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:15.374651 21025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:15.374924 21025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:15.377821 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "b078f666f94740d3ba8064301f1de5ee"
format_stamp: "Formatted at 2026-08-06 22:46:15 on dist-test-slave-zr1t"
server_key: "c29972cb5f8770155688746927bb5bb0"
server_key_iv: "e189c14498912f5b1b7f253756bf54fa"
server_key_version: "encryptionkey@0"
I20260806 22:46:15.378604 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "b078f666f94740d3ba8064301f1de5ee"
format_stamp: "Formatted at 2026-08-06 22:46:15 on dist-test-slave-zr1t"
server_key: "c29972cb5f8770155688746927bb5bb0"
server_key_iv: "e189c14498912f5b1b7f253756bf54fa"
server_key_version: "encryptionkey@0"
I20260806 22:46:15.384275 21025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 22:46:15.387907 21046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:15.389318 21025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:46:15.389587 21025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "b078f666f94740d3ba8064301f1de5ee"
format_stamp: "Formatted at 2026-08-06 22:46:15 on dist-test-slave-zr1t"
server_key: "c29972cb5f8770155688746927bb5bb0"
server_key_iv: "e189c14498912f5b1b7f253756bf54fa"
server_key_version: "encryptionkey@0"
I20260806 22:46:15.389820 21025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:15.405764 21025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:15.409787 21025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:15.410029 21025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:15.410954 21025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:15.412207 21025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:15.412288 21025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:15.412376 21025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:15.412429 21025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:15.447036 21025 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:32865
I20260806 22:46:15.447089 21212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:32865 every 8 connection(s)
I20260806 22:46:15.448341 21025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:46:15.455209 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 21025
I20260806 22:46:15.455426 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:46:15.455900 23954 external_mini_cluster.cc:1428] Setting key e8b358e175ad5a3f7ca25e430d91719a
Aug 06 22:46:15 dist-test-slave-zr1t krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056375, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.130@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:46:15.470546 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:15.450973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:54965 (local address 127.23.100.190:43289)
0806 22:46:15.451483 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:15.451488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:15.452452 (+ 964us) server_negotiation.cc:407] Connection header received
0806 22:46:15.454388 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:15.454393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:15.454485 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:15.454608 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:15.456255 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:15.457150 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:15.458839 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:15.459269 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:15.459417 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:15.463498 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:15.463507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:15.463514 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:15.463557 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:15.467037 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:15.468026 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:15.468034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:15.468039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:15.468184 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:15.468699 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:15.468706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:15.468711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:15.469046 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:15.469181 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:15.470074 (+ 893us) server_negotiation.cc:299] Negotiation successful
0806 22:46:15.470252 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":279,"threads_started":1}
I20260806 22:46:15.472038 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:15.451543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:43289 (local address 127.23.100.130:54965)
0806 22:46:15.452265 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:15.452302 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:15.454105 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:15.454863 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:15.454878 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:15.455449 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:15.456088 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:15.456106 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:15.457312 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:15.457317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:15.458688 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:15.458700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:15.459491 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:15.459500 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:15.459746 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:15.460442 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:46:15.460464 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:15.463200 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:15.467194 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:15.467206 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:15.467226 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:15.467895 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:15.468299 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:15.468306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:15.468312 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:15.468579 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:15.469178 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:15.469189 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:15.469669 (+ 480us) client_negotiation.cc:769] Sending connection context
0806 22:46:15.469985 (+ 316us) client_negotiation.cc:240] Negotiation successful
0806 22:46:15.470570 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":289,"threads_started":1}
I20260806 22:46:15.473591 21213 heartbeater.cc:347] Connected to a master server at 127.23.100.190:43289
I20260806 22:46:15.473971 21213 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:15.474882 21213 heartbeater.cc:511] Master 127.23.100.190:43289 requested a full tablet report, sending...
I20260806 22:46:15.476487 20753 ts_manager.cc:194] Registered new tserver with Master: b078f666f94740d3ba8064301f1de5ee (127.23.100.130:32865)
I20260806 22:46:15.477339 20753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.130@KRBTEST.COM'} at 127.23.100.130:54965
WARNING: no policy specified for kudu/127.23.100.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
Entry for principal kudu/127.23.100.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab.
I20260806 22:46:15.529335 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43289
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:15.871181 21225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:15.871538 21225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:15.871719 21225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:15.880275 21225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:15.880434 21225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:15.880625 21225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:46:15.890807 21225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.131.keytab
--principal=kudu/127.23.100.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:43289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:15.892146 21225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:15.893553 21225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:15.907823 21233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:15.907904 21235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:15.907941 21238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:15.908883 21225 server_base.cc:1034] running on GCE node
I20260806 22:46:15.909411 21225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:15.910394 21225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:15.911623 21225 hybrid_clock.cc:648] HybridClock initialized: now 1786056375911586 us; error 48 us; skew 500 ppm
I20260806 22:46:15.911952 21225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:15 dist-test-slave-zr1t krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056375, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:15.915099 21225 init.cc:377] Logged in from keytab as kudu/127.23.100.131@KRBTEST.COM (short username kudu)
I20260806 22:46:15.916512 21225 webserver.cc:460] Webserver started at http://127.23.100.131:39335/ using document root <none> and password file <none>
I20260806 22:46:15.917191 21225 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:15.917342 21225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:15.917722 21225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:15.920558 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "43dc8c3114a84e538b7f674f03dc211f"
format_stamp: "Formatted at 2026-08-06 22:46:15 on dist-test-slave-zr1t"
server_key: "a288187a8ad79963dce7d21806b6f391"
server_key_iv: "e623db92de1ae5b8cff68ebd260bbfa1"
server_key_version: "encryptionkey@0"
I20260806 22:46:15.921331 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "43dc8c3114a84e538b7f674f03dc211f"
format_stamp: "Formatted at 2026-08-06 22:46:15 on dist-test-slave-zr1t"
server_key: "a288187a8ad79963dce7d21806b6f391"
server_key_iv: "e623db92de1ae5b8cff68ebd260bbfa1"
server_key_version: "encryptionkey@0"
I20260806 22:46:15.926514 21225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:46:15.929879 21013 heartbeater.cc:503] Master 127.23.100.190:43289 was elected leader, sending a full tablet report...
I20260806 22:46:15.929999 21246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:15.931370 21225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:46:15.931543 21225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "43dc8c3114a84e538b7f674f03dc211f"
format_stamp: "Formatted at 2026-08-06 22:46:15 on dist-test-slave-zr1t"
server_key: "a288187a8ad79963dce7d21806b6f391"
server_key_iv: "e623db92de1ae5b8cff68ebd260bbfa1"
server_key_version: "encryptionkey@0"
I20260806 22:46:15.931739 21225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:15.945930 21225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:15.950505 21225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:15.950798 21225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:15.951776 21225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:15.953136 21225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:15.953243 21225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:15.953388 21225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:15.953474 21225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:15.987505 21225 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:43361
I20260806 22:46:15.987545 21421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:43361 every 8 connection(s)
I20260806 22:46:15.988765 21225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:46:15.991999 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 21225
I20260806 22:46:15.992202 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:46:15.992781 23954 external_mini_cluster.cc:1428] Setting key 88a23250a0fdb349f6cdf8322c9cd9bb
Aug 06 22:46:15 dist-test-slave-zr1t krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056375, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.131@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:46:16.010255 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:15.991440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:38611 (local address 127.23.100.190:43289)
0806 22:46:15.991947 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:15.991952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:15.993065 (+ 1113us) server_negotiation.cc:407] Connection header received
0806 22:46:15.995387 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:15.995393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:15.995513 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:15.995629 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:15.997302 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:15.998263 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:15.999802 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.000216 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.000322 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.003984 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:16.003989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:16.003992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:16.004025 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:16.007016 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:16.007940 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:16.007944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:16.007947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:16.008137 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:16.008597 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:16.008601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:16.008603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:16.008899 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:16.009021 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:16.009801 (+ 780us) server_negotiation.cc:299] Negotiation successful
0806 22:46:16.009988 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":272,"threads_started":1}
I20260806 22:46:16.011225 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:15.991942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:43289 (local address 127.23.100.131:38611)
0806 22:46:15.992859 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:15.992912 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:15.995105 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:15.995902 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:15.995915 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:15.996492 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:15.997126 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:15.997144 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:15.998401 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:15.998405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:15.999637 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:15.999649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.000498 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.000508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:16.000736 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.001488 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:46:16.001512 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:16.003784 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:16.007168 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:16.007175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:16.007188 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:16.007813 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:16.008264 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:46:16.008270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:46:16.008276 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:46:16.008491 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:46:16.009042 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:16.009049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:16.009360 (+ 311us) client_negotiation.cc:769] Sending connection context
0806 22:46:16.009699 (+ 339us) client_negotiation.cc:240] Negotiation successful
0806 22:46:16.010092 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":380,"threads_started":1}
I20260806 22:46:16.012557 21422 heartbeater.cc:347] Connected to a master server at 127.23.100.190:43289
I20260806 22:46:16.012840 21422 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:16.013492 21422 heartbeater.cc:511] Master 127.23.100.190:43289 requested a full tablet report, sending...
I20260806 22:46:16.014938 20753 ts_manager.cc:194] Registered new tserver with Master: 43dc8c3114a84e538b7f674f03dc211f (127.23.100.131:43361)
I20260806 22:46:16.015775 20753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.100.131@KRBTEST.COM'} at 127.23.100.131:38611
I20260806 22:46:16.020545 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:46:16.038401 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.024984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.23.100.190:43289)
0806 22:46:16.025163 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:16.025168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:16.025599 (+ 431us) server_negotiation.cc:407] Connection header received
0806 22:46:16.025776 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:16.025780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:16.025844 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:16.025928 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:16.027458 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.028179 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.029660 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.030123 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.030231 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.032698 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:16.032704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:16.032707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:16.032740 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:16.035603 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:16.036571 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:16.036576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:16.036578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:16.036703 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:16.037069 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:16.037073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:16.037075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:16.037401 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:16.037661 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:16.037963 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 22:46:16.038144 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 22:46:16.042568 20753 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:46:16.053217 20753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:46:16.083789 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.070037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.23.100.131:43361)
0806 22:46:16.070940 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:16.070950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:16.074266 (+ 3316us) server_negotiation.cc:407] Connection header received
0806 22:46:16.074676 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:16.074689 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:16.075021 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:16.075239 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:16.076578 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.077995 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.080537 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.081467 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.081719 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.081923 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:16.083235 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0806 22:46:16.083418 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":479,"threads_started":1}
I20260806 22:46:16.085013 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.069252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:43361 (local address 127.0.0.1:52848)
0806 22:46:16.074100 (+ 4848us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:16.074125 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:16.074412 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:16.075723 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:16.075735 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:16.075782 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:16.076326 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:16.076341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.078192 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.078201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:16.080357 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:16.080380 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.082084 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.082090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:16.082942 (+ 852us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.083018 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 22:46:16.084363 (+ 1345us) client_negotiation.cc:240] Negotiation successful
0806 22:46:16.084572 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4673,"mutex_wait_us":28,"thread_start_us":4490,"threads_started":1}
I20260806 22:46:16.086618 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.068833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:46015 (local address 127.0.0.1:53480)
0806 22:46:16.069665 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:16.069711 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:16.069930 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:16.077816 (+ 7886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:16.077822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:16.077855 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:16.078351 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:16.078369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.081629 (+ 3260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.081635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:16.083809 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:16.083834 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.085816 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.085823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:16.085992 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.086024 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 22:46:16.086089 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 22:46:16.086208 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":23,"thread_start_us":344,"threads_started":1}
I20260806 22:46:16.087265 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.074485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.23.100.130:32865)
0806 22:46:16.075139 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:16.075148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:16.075883 (+ 735us) server_negotiation.cc:407] Connection header received
0806 22:46:16.076227 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:16.076235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:16.076464 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:16.076650 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:16.077623 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.079543 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.081998 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.084622 (+ 2624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.084841 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.085073 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:16.085941 (+ 868us) server_negotiation.cc:299] Negotiation successful
0806 22:46:16.086176 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":364,"threads_started":1}
I20260806 22:46:16.088171 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.075182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:32865 (local address 127.0.0.1:44544)
0806 22:46:16.075763 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:16.075781 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:16.076012 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:16.076942 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:16.076946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:16.076971 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:16.077406 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:16.077420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.079665 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.079670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:16.081834 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:16.081849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.085241 (+ 3392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.085251 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:16.085500 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.085595 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 22:46:16.085704 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 22:46:16.087707 (+ 2003us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":304,"threads_started":1}
I20260806 22:46:16.088194 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.069462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.23.100.129:46015)
0806 22:46:16.077105 (+ 7643us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:16.077116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:16.077145 (+ 29us) server_negotiation.cc:407] Connection header received
0806 22:46:16.077255 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:16.077263 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:16.077582 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:16.078613 (+ 1031us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:16.079839 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.081439 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.084022 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.085551 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.087229 (+ 1678us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.087604 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:16.087680 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 22:46:16.087871 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7430,"thread_start_us":7233,"threads_started":1}
I20260806 22:46:16.089648 21326 tablet_service.cc:1467] Processing CreateTablet for tablet bcb8b772f35344dfa69bb324fc600793 (DEFAULT_TABLE table=test-table [id=fab36fdf25054fa4b08c66e1ff16c7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:16.093868 20929 tablet_service.cc:1467] Processing CreateTablet for tablet bcb8b772f35344dfa69bb324fc600793 (DEFAULT_TABLE table=test-table [id=fab36fdf25054fa4b08c66e1ff16c7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:16.093868 21120 tablet_service.cc:1467] Processing CreateTablet for tablet bcb8b772f35344dfa69bb324fc600793 (DEFAULT_TABLE table=test-table [id=fab36fdf25054fa4b08c66e1ff16c7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:16.095355 20929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb8b772f35344dfa69bb324fc600793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:16.095366 21120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb8b772f35344dfa69bb324fc600793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:16.098124 21326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb8b772f35344dfa69bb324fc600793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:16.111243 21453 tablet_bootstrap.cc:492] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: Bootstrap starting.
I20260806 22:46:16.111488 21455 tablet_bootstrap.cc:492] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: Bootstrap starting.
I20260806 22:46:16.112259 21454 tablet_bootstrap.cc:492] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: Bootstrap starting.
I20260806 22:46:16.115538 21453 tablet_bootstrap.cc:654] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:16.115705 21454 tablet_bootstrap.cc:654] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:16.116122 21455 tablet_bootstrap.cc:654] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:16.116878 21454 log.cc:824] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:16.116891 21453 log.cc:824] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:16.117177 21455 log.cc:824] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:16.119817 21453 tablet_bootstrap.cc:492] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: No bootstrap required, opened a new log
I20260806 22:46:16.120090 21453 ts_tablet_manager.cc:1397] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 22:46:16.120285 21454 tablet_bootstrap.cc:492] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: No bootstrap required, opened a new log
I20260806 22:46:16.120285 21455 tablet_bootstrap.cc:492] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: No bootstrap required, opened a new log
I20260806 22:46:16.120548 21455 ts_tablet_manager.cc:1397] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260806 22:46:16.120551 21454 ts_tablet_manager.cc:1397] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260806 22:46:16.127943 21453 raft_consensus.cc:348] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa00671e49854dfc8683872990595dc9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46015 } } peers { permanent_uuid: "43dc8c3114a84e538b7f674f03dc211f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43361 } } peers { permanent_uuid: "b078f666f94740d3ba8064301f1de5ee" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 32865 } }
I20260806 22:46:16.128329 21453 raft_consensus.cc:374] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:16.128436 21453 raft_consensus.cc:729] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43dc8c3114a84e538b7f674f03dc211f, State: Initialized, Role: FOLLOWER
I20260806 22:46:16.129058 21453 consensus_queue.cc:260] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa00671e49854dfc8683872990595dc9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46015 } } peers { permanent_uuid: "43dc8c3114a84e538b7f674f03dc211f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43361 } } peers { permanent_uuid: "b078f666f94740d3ba8064301f1de5ee" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 32865 } }
I20260806 22:46:16.130162 21422 heartbeater.cc:503] Master 127.23.100.190:43289 was elected leader, sending a full tablet report...
I20260806 22:46:16.131384 21455 raft_consensus.cc:348] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa00671e49854dfc8683872990595dc9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46015 } } peers { permanent_uuid: "43dc8c3114a84e538b7f674f03dc211f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43361 } } peers { permanent_uuid: "b078f666f94740d3ba8064301f1de5ee" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 32865 } }
I20260806 22:46:16.131381 21454 raft_consensus.cc:348] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa00671e49854dfc8683872990595dc9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46015 } } peers { permanent_uuid: "43dc8c3114a84e538b7f674f03dc211f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43361 } } peers { permanent_uuid: "b078f666f94740d3ba8064301f1de5ee" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 32865 } }
I20260806 22:46:16.131881 21455 raft_consensus.cc:374] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:16.132046 21453 ts_tablet_manager.cc:1428] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260806 22:46:16.132026 21455 raft_consensus.cc:729] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa00671e49854dfc8683872990595dc9, State: Initialized, Role: FOLLOWER
I20260806 22:46:16.132490 21454 raft_consensus.cc:374] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:16.132654 21454 raft_consensus.cc:729] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b078f666f94740d3ba8064301f1de5ee, State: Initialized, Role: FOLLOWER
I20260806 22:46:16.132771 21455 consensus_queue.cc:260] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa00671e49854dfc8683872990595dc9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46015 } } peers { permanent_uuid: "43dc8c3114a84e538b7f674f03dc211f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43361 } } peers { permanent_uuid: "b078f666f94740d3ba8064301f1de5ee" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 32865 } }
I20260806 22:46:16.133404 21454 consensus_queue.cc:260] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa00671e49854dfc8683872990595dc9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46015 } } peers { permanent_uuid: "43dc8c3114a84e538b7f674f03dc211f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43361 } } peers { permanent_uuid: "b078f666f94740d3ba8064301f1de5ee" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 32865 } }
I20260806 22:46:16.134817 21455 ts_tablet_manager.cc:1428] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260806 22:46:16.135844 21213 heartbeater.cc:503] Master 127.23.100.190:43289 was elected leader, sending a full tablet report...
I20260806 22:46:16.136490 21454 ts_tablet_manager.cc:1428] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260806 22:46:16.150713 21014 tablet.cc:2369] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:46:16.200364 21215 tablet.cc:2369] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:46:16.240453 21423 tablet.cc:2369] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:46:16.342437 21462 raft_consensus.cc:482] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:46:16.342767 21462 raft_consensus.cc:504] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa00671e49854dfc8683872990595dc9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46015 } } peers { permanent_uuid: "43dc8c3114a84e538b7f674f03dc211f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43361 } } peers { permanent_uuid: "b078f666f94740d3ba8064301f1de5ee" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 32865 } }
I20260806 22:46:16.344483 21462 leader_election.cc:288] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43dc8c3114a84e538b7f674f03dc211f (127.23.100.131:43361), b078f666f94740d3ba8064301f1de5ee (127.23.100.130:32865)
I20260806 22:46:16.351255 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.345007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:41381 (local address 127.23.100.131:43361)
0806 22:46:16.345218 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:16.345223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:16.345864 (+ 641us) server_negotiation.cc:407] Connection header received
0806 22:46:16.346150 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:16.346159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:16.346251 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:16.346317 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:16.347043 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.347798 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.349566 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.350325 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.350509 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.350592 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:16.350868 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 22:46:16.350996 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:46:16.351320 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.344905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:43361 (local address 127.23.100.129:41381)
0806 22:46:16.345707 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:16.345729 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:16.345911 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:16.346517 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:16.346521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:16.346555 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:16.346887 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:16.346895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.347928 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.347937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:16.349371 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:16.349383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.350540 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.350546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:16.350705 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.350729 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 22:46:16.350839 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 22:46:16.350977 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":416,"threads_started":1}
I20260806 22:46:16.352303 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.345871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:32865 (local address 127.23.100.129:59289)
0806 22:46:16.346570 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:16.346596 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:16.346794 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:16.347175 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:16.347179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:16.347206 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:16.347586 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:16.347595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.348713 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.348719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:16.350307 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:16.350321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.351629 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.351638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:16.351815 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.351848 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 22:46:16.351960 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 22:46:16.352083 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":314,"threads_started":1}
I20260806 22:46:16.352449 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.346262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:59289 (local address 127.23.100.130:32865)
0806 22:46:16.346501 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:16.346508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:16.346705 (+ 197us) server_negotiation.cc:407] Connection header received
0806 22:46:16.346960 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:16.346967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:16.347038 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:16.347150 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:16.347779 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.348550 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.350489 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.351356 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.351544 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.351611 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:16.352037 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 22:46:16.352189 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 22:46:16.352555 21355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb8b772f35344dfa69bb324fc600793" candidate_uuid: "fa00671e49854dfc8683872990595dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43dc8c3114a84e538b7f674f03dc211f" is_pre_election: true
I20260806 22:46:16.353159 21355 raft_consensus.cc:2454] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa00671e49854dfc8683872990595dc9 in term 0.
I20260806 22:46:16.353415 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb8b772f35344dfa69bb324fc600793" candidate_uuid: "fa00671e49854dfc8683872990595dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b078f666f94740d3ba8064301f1de5ee" is_pre_election: true
I20260806 22:46:16.353982 21151 raft_consensus.cc:2454] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa00671e49854dfc8683872990595dc9 in term 0.
I20260806 22:46:16.354023 20867 leader_election.cc:302] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43dc8c3114a84e538b7f674f03dc211f, fa00671e49854dfc8683872990595dc9; no voters:
I20260806 22:46:16.354480 21462 raft_consensus.cc:2781] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:46:16.354624 21462 raft_consensus.cc:482] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:46:16.354748 21462 raft_consensus.cc:3037] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:16.357008 21462 raft_consensus.cc:504] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa00671e49854dfc8683872990595dc9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46015 } } peers { permanent_uuid: "43dc8c3114a84e538b7f674f03dc211f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43361 } } peers { permanent_uuid: "b078f666f94740d3ba8064301f1de5ee" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 32865 } }
I20260806 22:46:16.357690 21462 leader_election.cc:288] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [CANDIDATE]: Term 1 election: Requested vote from peers 43dc8c3114a84e538b7f674f03dc211f (127.23.100.131:43361), b078f666f94740d3ba8064301f1de5ee (127.23.100.130:32865)
I20260806 22:46:16.358178 21355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb8b772f35344dfa69bb324fc600793" candidate_uuid: "fa00671e49854dfc8683872990595dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43dc8c3114a84e538b7f674f03dc211f"
I20260806 22:46:16.358265 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb8b772f35344dfa69bb324fc600793" candidate_uuid: "fa00671e49854dfc8683872990595dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b078f666f94740d3ba8064301f1de5ee"
I20260806 22:46:16.358405 21355 raft_consensus.cc:3037] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:16.358464 21151 raft_consensus.cc:3037] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:16.360684 21151 raft_consensus.cc:2454] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa00671e49854dfc8683872990595dc9 in term 1.
I20260806 22:46:16.360863 21355 raft_consensus.cc:2454] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa00671e49854dfc8683872990595dc9 in term 1.
I20260806 22:46:16.361162 20867 leader_election.cc:302] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b078f666f94740d3ba8064301f1de5ee, fa00671e49854dfc8683872990595dc9; no voters:
I20260806 22:46:16.361490 21462 raft_consensus.cc:2781] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:16.361981 21462 raft_consensus.cc:686] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 1 LEADER]: Becoming Leader. State: Replica: fa00671e49854dfc8683872990595dc9, State: Running, Role: LEADER
I20260806 22:46:16.362499 21462 consensus_queue.cc:237] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa00671e49854dfc8683872990595dc9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46015 } } peers { permanent_uuid: "43dc8c3114a84e538b7f674f03dc211f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43361 } } peers { permanent_uuid: "b078f666f94740d3ba8064301f1de5ee" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 32865 } }
I20260806 22:46:16.367501 20753 catalog_manager.cc:5495] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa00671e49854dfc8683872990595dc9 (127.23.100.129). New cstate: current_term: 1 leader_uuid: "fa00671e49854dfc8683872990595dc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa00671e49854dfc8683872990595dc9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 46015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43dc8c3114a84e538b7f674f03dc211f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b078f666f94740d3ba8064301f1de5ee" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 32865 } health_report { overall_health: UNKNOWN } } }
I20260806 22:46:16.388957 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.383466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.23.100.129:46015)
0806 22:46:16.383657 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:16.383661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:16.383756 (+ 95us) server_negotiation.cc:407] Connection header received
0806 22:46:16.383942 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:16.383946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:16.384041 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:16.384169 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:46:16.384511 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.385212 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.386761 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.387243 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.387335 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.387897 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:46:16.388404 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:46:16.388512 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:16.388665 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 22:46:16.388732 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 22:46:16.397588 21151 raft_consensus.cc:1261] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee [term 1 FOLLOWER]: Refusing update from remote peer fa00671e49854dfc8683872990595dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:16.397647 21355 raft_consensus.cc:1261] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f [term 1 FOLLOWER]: Refusing update from remote peer fa00671e49854dfc8683872990595dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:16.398641 21467 consensus_queue.cc:1035] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b078f666f94740d3ba8064301f1de5ee" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:46:16.399008 21462 consensus_queue.cc:1035] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43dc8c3114a84e538b7f674f03dc211f" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 43361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:46:16.408733 21474 mvcc.cc:205] Tried to move back new op lower bound from 7315686917714067456 to 7315686917584789504. Current Snapshot: MvccSnapshot[applied={T|T < 7315686917714067456}]
I20260806 22:46:16.410089 21476 mvcc.cc:205] Tried to move back new op lower bound from 7315686917714067456 to 7315686917584789504. Current Snapshot: MvccSnapshot[applied={T|T < 7315686917714067456}]
I20260806 22:46:16.432196 20753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:16.432493 20753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:16.436245 20753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edeb600022f8423dbf48b3f387bbded8: Sending DeleteTablet for 3 replicas of tablet bcb8b772f35344dfa69bb324fc600793
I20260806 22:46:16.437415 21326 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb8b772f35344dfa69bb324fc600793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:16 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:52848
I20260806 22:46:16.437427 20929 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb8b772f35344dfa69bb324fc600793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:16 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:53480
I20260806 22:46:16.439059 21120 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb8b772f35344dfa69bb324fc600793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:16 UTC) from {username='kudu', principal='kudu/127.23.100.190@KRBTEST.COM'} at 127.0.0.1:44544
I20260806 22:46:16.440359 21492 tablet_replica.cc:330] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: stopping tablet replica
I20260806 22:46:16.440806 21492 raft_consensus.cc:2229] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:16.441325 21492 raft_consensus.cc:2258] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:16.442037 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 20839
I20260806 22:46:16.443710 21492 ts_tablet_manager.cc:1905] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:16.449968 21490 tablet_replica.cc:330] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: stopping tablet replica
I20260806 22:46:16.450383 21490 raft_consensus.cc:2229] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:16.450903 21490 raft_consensus.cc:2258] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:16.451851 21491 tablet_replica.cc:330] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: stopping tablet replica
I20260806 22:46:16.452251 21491 raft_consensus.cc:2229] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:46:16.452929 21491 raft_consensus.cc:2258] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:16.453203 21490 ts_tablet_manager.cc:1905] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:16.454799 21492 ts_tablet_manager.cc:1918] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:16.455006 21492 log.cc:1196] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/bcb8b772f35344dfa69bb324fc600793
I20260806 22:46:16.455579 21492 ts_tablet_manager.cc:1939] T bcb8b772f35344dfa69bb324fc600793 P b078f666f94740d3ba8064301f1de5ee: Deleting consensus metadata
I20260806 22:46:16.457547 20730 catalog_manager.cc:4841] TS b078f666f94740d3ba8064301f1de5ee (127.23.100.130:32865): tablet bcb8b772f35344dfa69bb324fc600793 (table test-table [id=fab36fdf25054fa4b08c66e1ff16c7f7]) successfully deleted
I20260806 22:46:16.458794 21491 ts_tablet_manager.cc:1905] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:16.461428 21490 ts_tablet_manager.cc:1918] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:16.461668 21490 log.cc:1196] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/bcb8b772f35344dfa69bb324fc600793
I20260806 22:46:16.462339 21490 ts_tablet_manager.cc:1939] T bcb8b772f35344dfa69bb324fc600793 P 43dc8c3114a84e538b7f674f03dc211f: Deleting consensus metadata
I20260806 22:46:16.464504 20730 catalog_manager.cc:4841] TS 43dc8c3114a84e538b7f674f03dc211f (127.23.100.131:43361): tablet bcb8b772f35344dfa69bb324fc600793 (table test-table [id=fab36fdf25054fa4b08c66e1ff16c7f7]) successfully deleted
I20260806 22:46:16.466200 21491 ts_tablet_manager.cc:1918] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:16.466372 21491 log.cc:1196] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/bcb8b772f35344dfa69bb324fc600793
I20260806 22:46:16.466908 21491 ts_tablet_manager.cc:1939] T bcb8b772f35344dfa69bb324fc600793 P fa00671e49854dfc8683872990595dc9: Deleting consensus metadata
I20260806 22:46:16.468638 20728 catalog_manager.cc:4841] TS fa00671e49854dfc8683872990595dc9 (127.23.100.129:46015): tablet bcb8b772f35344dfa69bb324fc600793 (table test-table [id=fab36fdf25054fa4b08c66e1ff16c7f7]) successfully deleted
Aug 06 22:46:16 dist-test-slave-zr1t krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.129@KRBTEST.COM
I20260806 22:46:16.491467 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.443756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.23.100.129:46015)
0806 22:46:16.444003 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:16.444012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:16.469395 (+ 25383us) server_negotiation.cc:407] Connection header received
0806 22:46:16.469611 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:16.469618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:16.469714 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:16.469825 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:16.471387 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.472646 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.474441 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.475156 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.475287 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.484091 (+ 8804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:16.484110 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:16.484135 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:16.484190 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:16.488354 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:16.489155 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:16.489166 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:16.489169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:16.489361 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:16.489794 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:16.489800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:16.489803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:16.490252 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:16.490577 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:16.490772 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 22:46:16.491027 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 22:46:16.492404 21006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:16.595669 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 20839
I20260806 22:46:16.624318 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 21025
Aug 06 22:46:16 dist-test-slave-zr1t krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.130@KRBTEST.COM
I20260806 22:46:16.641088 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.625164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.23.100.130:32865)
0806 22:46:16.625418 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:16.625424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:16.625447 (+ 23us) server_negotiation.cc:407] Connection header received
0806 22:46:16.625704 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:16.625709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:16.625790 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:16.625919 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:16.627295 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.628062 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.629383 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.629945 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.630096 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.634876 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:16.634895 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:16.634916 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:16.634960 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:16.638397 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:16.639147 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:16.639154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:16.639157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:16.639276 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:16.639642 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:16.639646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:16.639648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:16.640014 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:16.640252 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:16.640530 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 22:46:16.640711 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 22:46:16.641726 21207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:16.732647 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 21025
I20260806 22:46:16.758037 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 21225
Aug 06 22:46:16 dist-test-slave-zr1t krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.131@KRBTEST.COM
I20260806 22:46:16.773866 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:16.758803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.23.100.131:43361)
0806 22:46:16.759015 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:16.759023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:16.759048 (+ 25us) server_negotiation.cc:407] Connection header received
0806 22:46:16.759267 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:16.759271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:16.759341 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:16.759453 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:16.760779 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.761826 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.763215 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:16.763686 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:16.763818 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:16.768406 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:16.768423 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:16.768439 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:16.768478 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:16.771350 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:16.771991 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:16.771997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:16.772000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:16.772143 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:16.772469 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:16.772472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:16.772474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:16.772813 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:16.773037 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:16.773288 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 22:46:16.773459 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 22:46:16.774530 21413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:16.864558 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 21225
I20260806 22:46:16.890707 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 20709
I20260806 22:46:16.891932 20798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:17.002612 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 20709
2026-08-06T22:46:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4995 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T22:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:17Z Disabled control of system clock
I20260806 22:46:17.113323 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:41399
--webserver_interface=127.23.100.190
--webserver_port=0
--builtin_ntp_servers=127.23.100.148:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:41399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 22:46:17.458838 21505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:17.459126 21505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:17.459264 21505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:17.466651 21505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:17.466760 21505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:17.466838 21505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:17.466931 21505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:17.467020 21505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:17.476207 21505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:41399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:17.477463 21505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:17.479270 21505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:17.491359 21512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:17.491709 21515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:17.492235 21513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:17.492369 21505 server_base.cc:1034] running on GCE node
I20260806 22:46:17.493191 21505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:17.494661 21505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:17.495944 21505 hybrid_clock.cc:648] HybridClock initialized: now 1786056377495893 us; error 63 us; skew 500 ppm
I20260806 22:46:17.496326 21505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:17.502629 21505 webserver.cc:460] Webserver started at http://127.23.100.190:43599/ using document root <none> and password file <none>
I20260806 22:46:17.503379 21505 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:17.503518 21505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:17.503888 21505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:17.507076 21505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "ffb2af92185244458fdac3d8adf58f5c"
format_stamp: "Formatted at 2026-08-06 22:46:17 on dist-test-slave-zr1t"
server_key: "0390957cc62a5c3cccdc0680e0bf6dad"
server_key_iv: "3a4c52a67a5d79c620500817ec1c781b"
server_key_version: "encryptionkey@0"
I20260806 22:46:17.507916 21505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "ffb2af92185244458fdac3d8adf58f5c"
format_stamp: "Formatted at 2026-08-06 22:46:17 on dist-test-slave-zr1t"
server_key: "0390957cc62a5c3cccdc0680e0bf6dad"
server_key_iv: "3a4c52a67a5d79c620500817ec1c781b"
server_key_version: "encryptionkey@0"
I20260806 22:46:17.513206 21505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 22:46:17.516654 21526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:17.518036 21505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 22:46:17.518256 21505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "ffb2af92185244458fdac3d8adf58f5c"
format_stamp: "Formatted at 2026-08-06 22:46:17 on dist-test-slave-zr1t"
server_key: "0390957cc62a5c3cccdc0680e0bf6dad"
server_key_iv: "3a4c52a67a5d79c620500817ec1c781b"
server_key_version: "encryptionkey@0"
I20260806 22:46:17.518534 21505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:17.539033 21505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:17.539919 21505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:17.540264 21505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:17.560997 21505 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:41399
I20260806 22:46:17.561033 21608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:41399 every 8 connection(s)
I20260806 22:46:17.563069 21505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:17.564579 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 21505
I20260806 22:46:17.564780 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:17.565374 23954 external_mini_cluster.cc:1428] Setting key 29babf56ec007616e6f62caaca954787
I20260806 22:46:17.568233 21609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:17.583091 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c: Bootstrap starting.
I20260806 22:46:17.587500 21609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:17.589007 21609 log.cc:824] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:17.592240 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c: No bootstrap required, opened a new log
I20260806 22:46:17.592546 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:17.568524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.23.100.190:41399)
0806 22:46:17.569838 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:17.569858 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:17.574107 (+ 4249us) server_negotiation.cc:407] Connection header received
0806 22:46:17.576937 (+ 2830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:17.576966 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:17.577612 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:17.579245 (+ 1633us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:17.579862 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:17.582073 (+ 2211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:17.585730 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:17.586895 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:17.587099 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:17.588625 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:17.588638 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:17.588658 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:17.589577 (+ 919us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:17.589766 (+ 189us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:17.589799 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:17.589938 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:17.590557 (+ 619us) server_negotiation.cc:299] Negotiation successful
0806 22:46:17.590961 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":438,"threads_started":1}
I20260806 22:46:17.604698 21609 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb2af92185244458fdac3d8adf58f5c" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41399 } }
I20260806 22:46:17.605218 21609 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:17.605352 21609 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb2af92185244458fdac3d8adf58f5c, State: Initialized, Role: FOLLOWER
I20260806 22:46:17.606113 21609 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb2af92185244458fdac3d8adf58f5c" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41399 } }
I20260806 22:46:17.606604 21609 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:17.606819 21609 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:17.607021 21609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:17.610908 21609 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb2af92185244458fdac3d8adf58f5c" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41399 } }
I20260806 22:46:17.611583 21609 leader_election.cc:302] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffb2af92185244458fdac3d8adf58f5c; no voters:
I20260806 22:46:17.612362 21609 leader_election.cc:288] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:17.612591 21615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:17.613422 21615 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [term 1 LEADER]: Becoming Leader. State: Replica: ffb2af92185244458fdac3d8adf58f5c, State: Running, Role: LEADER
I20260806 22:46:17.614025 21615 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb2af92185244458fdac3d8adf58f5c" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41399 } }
I20260806 22:46:17.614863 21609 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:17.617797 21616 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffb2af92185244458fdac3d8adf58f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb2af92185244458fdac3d8adf58f5c" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41399 } } }
I20260806 22:46:17.617687 21619 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffb2af92185244458fdac3d8adf58f5c. Latest consensus state: current_term: 1 leader_uuid: "ffb2af92185244458fdac3d8adf58f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffb2af92185244458fdac3d8adf58f5c" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 41399 } } }
I20260806 22:46:17.618463 21616 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:17.618525 21619 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:17.619799 21629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:17.633883 21629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:17.643975 21629 catalog_manager.cc:1347] Generated new cluster ID: 9beda9af68c84663bd506600550fb7e1
I20260806 22:46:17.644111 21629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:17.654587 21629 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:17.655805 21629 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:17.663532 21629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c: Generated new TSK 0
I20260806 22:46:17.664328 21629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:17.678341 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.129:0
--local_ip_for_outbound_sockets=127.23.100.129
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41399
--builtin_ntp_servers=127.23.100.148:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 22:46:18.029384 21643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:18.029754 21643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:18.029937 21643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:18.037353 21643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:18.037462 21643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:18.037659 21643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.129
I20260806 22:46:18.047008 21643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.100.129
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.129
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:18.048312 21643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:18.050103 21643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:18.063866 21655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:18.063898 21652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:18.063815 21651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:18.064608 21643 server_base.cc:1034] running on GCE node
I20260806 22:46:18.065188 21643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:18.066160 21643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:18.067364 21643 hybrid_clock.cc:648] HybridClock initialized: now 1786056378067323 us; error 28 us; skew 500 ppm
I20260806 22:46:18.067644 21643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:18.069905 21643 webserver.cc:460] Webserver started at http://127.23.100.129:36853/ using document root <none> and password file <none>
I20260806 22:46:18.070555 21643 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:18.070645 21643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:18.070895 21643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:18.073691 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/data/instance:
uuid: "9343103c26b8423889dae3d9c59585d9"
format_stamp: "Formatted at 2026-08-06 22:46:18 on dist-test-slave-zr1t"
server_key: "296d169d0620929d0bd80e9cacf2b99a"
server_key_iv: "c1d83dea2137d7a539af77d8ca9c11d4"
server_key_version: "encryptionkey@0"
I20260806 22:46:18.074430 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance:
uuid: "9343103c26b8423889dae3d9c59585d9"
format_stamp: "Formatted at 2026-08-06 22:46:18 on dist-test-slave-zr1t"
server_key: "296d169d0620929d0bd80e9cacf2b99a"
server_key_iv: "c1d83dea2137d7a539af77d8ca9c11d4"
server_key_version: "encryptionkey@0"
I20260806 22:46:18.079239 21643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:46:18.082456 21665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:18.083716 21643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:46:18.083902 21643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/wal
uuid: "9343103c26b8423889dae3d9c59585d9"
format_stamp: "Formatted at 2026-08-06 22:46:18 on dist-test-slave-zr1t"
server_key: "296d169d0620929d0bd80e9cacf2b99a"
server_key_iv: "c1d83dea2137d7a539af77d8ca9c11d4"
server_key_version: "encryptionkey@0"
I20260806 22:46:18.084112 21643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:18.103559 21643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:18.104388 21643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:18.104710 21643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:18.105736 21643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:18.107219 21643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:18.107477 21643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:18.107767 21643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:18.107882 21643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:18.146270 21643 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.129:45097
I20260806 22:46:18.146370 21833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.129:45097 every 8 connection(s)
I20260806 22:46:18.147722 21643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/data/info.pb
I20260806 22:46:18.153971 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 21643
I20260806 22:46:18.154172 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/wal/instance
I20260806 22:46:18.154646 23954 external_mini_cluster.cc:1428] Setting key 03473cb72c0ab8b721f224b686d893b0
I20260806 22:46:18.157786 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.130:0
--local_ip_for_outbound_sockets=127.23.100.130
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41399
--builtin_ntp_servers=127.23.100.148:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 22:46:18.161453 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:18.150417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.129:45653 (local address 127.23.100.190:41399)
0806 22:46:18.150939 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:18.150944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:18.152100 (+ 1156us) server_negotiation.cc:407] Connection header received
0806 22:46:18.154068 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:18.154075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:18.154175 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:18.154317 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:18.155656 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:18.156529 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:18.158413 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:18.158855 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:18.159004 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:18.160498 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:18.160504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:18.160507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:18.160525 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:18.160571 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:18.160584 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:18.160690 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:18.160951 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 22:46:18.161057 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":247,"threads_started":1}
I20260806 22:46:18.162725 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:18.150974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41399 (local address 127.23.100.129:45653)
0806 22:46:18.151922 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:18.151960 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:18.153771 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:18.154554 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:18.154562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:18.154651 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:18.155463 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:18.155494 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:18.156691 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:18.156698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:18.158252 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:18.158265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:18.159099 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:18.159111 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:18.159374 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:18.160147 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:46:18.160150 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:18.160190 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:46:18.160274 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:18.160691 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:18.160707 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:18.160718 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 22:46:18.160933 (+ 215us) client_negotiation.cc:240] Negotiation successful
0806 22:46:18.161360 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":288,"threads_started":1}
I20260806 22:46:18.164412 21837 heartbeater.cc:347] Connected to a master server at 127.23.100.190:41399
I20260806 22:46:18.164767 21837 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:18.165833 21837 heartbeater.cc:511] Master 127.23.100.190:41399 requested a full tablet report, sending...
I20260806 22:46:18.168521 21551 ts_manager.cc:194] Registered new tserver with Master: 9343103c26b8423889dae3d9c59585d9 (127.23.100.129:45097)
I20260806 22:46:18.170926 21551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.129:45653
W20260806 22:46:18.497189 21843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:18.497535 21843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:18.497735 21843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:18.504920 21843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:18.505069 21843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:18.505256 21843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.130
I20260806 22:46:18.514751 21843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.100.130
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.130
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:18.516098 21843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:18.517926 21843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:18.531606 21853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:18.531688 21854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:18.531658 21856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:18.532948 21843 server_base.cc:1034] running on GCE node
I20260806 22:46:18.533664 21843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:18.534694 21843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:18.535914 21843 hybrid_clock.cc:648] HybridClock initialized: now 1786056378535858 us; error 40 us; skew 500 ppm
I20260806 22:46:18.536243 21843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:18.538765 21843 webserver.cc:460] Webserver started at http://127.23.100.130:44187/ using document root <none> and password file <none>
I20260806 22:46:18.539477 21843 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:18.539597 21843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:18.539944 21843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:18.542977 21843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/data/instance:
uuid: "bb8b8dcf038c4284a788b59946901ec1"
format_stamp: "Formatted at 2026-08-06 22:46:18 on dist-test-slave-zr1t"
server_key: "ee0e8a9c4b23d94d974f07e8ad8ebcec"
server_key_iv: "b0cd6b5d53589807fae67eca59665921"
server_key_version: "encryptionkey@0"
I20260806 22:46:18.543797 21843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance:
uuid: "bb8b8dcf038c4284a788b59946901ec1"
format_stamp: "Formatted at 2026-08-06 22:46:18 on dist-test-slave-zr1t"
server_key: "ee0e8a9c4b23d94d974f07e8ad8ebcec"
server_key_iv: "b0cd6b5d53589807fae67eca59665921"
server_key_version: "encryptionkey@0"
I20260806 22:46:18.548933 21843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:46:18.552560 21865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:18.553999 21843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:46:18.554177 21843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/wal
uuid: "bb8b8dcf038c4284a788b59946901ec1"
format_stamp: "Formatted at 2026-08-06 22:46:18 on dist-test-slave-zr1t"
server_key: "ee0e8a9c4b23d94d974f07e8ad8ebcec"
server_key_iv: "b0cd6b5d53589807fae67eca59665921"
server_key_version: "encryptionkey@0"
I20260806 22:46:18.554378 21843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:18.576007 21843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:18.576800 21843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:18.577108 21843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:18.578181 21843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:18.579665 21843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:18.579754 21843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:18.579911 21843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:18.580005 21843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:18.614678 21843 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.130:45833
I20260806 22:46:18.614766 22016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.130:45833 every 8 connection(s)
I20260806 22:46:18.615963 21843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/data/info.pb
I20260806 22:46:18.623674 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 21843
I20260806 22:46:18.623883 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/wal/instance
I20260806 22:46:18.624374 23954 external_mini_cluster.cc:1428] Setting key c424a0b66109f367bd652dc287a496c6
I20260806 22:46:18.627444 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.100.131:0
--local_ip_for_outbound_sockets=127.23.100.131
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41399
--builtin_ntp_servers=127.23.100.148:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 22:46:18.629006 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:18.618662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:49347 (local address 127.23.100.190:41399)
0806 22:46:18.618855 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:18.618859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:18.620257 (+ 1398us) server_negotiation.cc:407] Connection header received
0806 22:46:18.622070 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:18.622075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:18.622145 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:18.622293 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:18.623291 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:18.624149 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:18.625987 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:18.626559 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:18.626661 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:18.628301 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:18.628306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:18.628309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:18.628326 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:18.628360 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:18.628373 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:18.628454 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:18.628708 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 22:46:18.628788 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 22:46:18.630614 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:18.619165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41399 (local address 127.23.100.130:49347)
0806 22:46:18.620103 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:18.620143 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:18.621820 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:18.622476 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:18.622484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:18.622567 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:18.623121 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:18.623136 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:18.624293 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:18.624300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:18.625842 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:18.625857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:18.626786 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:18.626794 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:18.627087 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:18.627916 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:46:18.627922 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:18.627972 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:46:18.628073 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:18.628481 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:18.628495 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:18.628506 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 22:46:18.628690 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 22:46:18.629093 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":345,"threads_started":1}
I20260806 22:46:18.632139 22018 heartbeater.cc:347] Connected to a master server at 127.23.100.190:41399
I20260806 22:46:18.632486 22018 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:18.633232 22018 heartbeater.cc:511] Master 127.23.100.190:41399 requested a full tablet report, sending...
I20260806 22:46:18.634714 21551 ts_manager.cc:194] Registered new tserver with Master: bb8b8dcf038c4284a788b59946901ec1 (127.23.100.130:45833)
I20260806 22:46:18.635546 21551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.130:49347
W20260806 22:46:18.989199 22023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:18.989490 22023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:18.989725 22023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:18.997475 22023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:18.997588 22023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:18.997748 22023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.100.131
I20260806 22:46:19.006784 22023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.100.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.100.131
--webserver_port=0
--tserver_master_addrs=127.23.100.190:41399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.100.131
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:19.008100 22023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:19.010004 22023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:19.023145 22028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:19.023314 22029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:19.023557 22032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:19.024164 22023 server_base.cc:1034] running on GCE node
I20260806 22:46:19.024780 22023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:19.025736 22023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:19.026957 22023 hybrid_clock.cc:648] HybridClock initialized: now 1786056379026905 us; error 27 us; skew 500 ppm
I20260806 22:46:19.027285 22023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 22:46:19.029604 22023 webserver.cc:460] Webserver started at http://127.23.100.131:36721/ using document root <none> and password file <none>
I20260806 22:46:19.030329 22023 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:19.030448 22023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:19.030774 22023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:19.033748 22023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/data/instance:
uuid: "3cfc564e8d0e40f8914eb683e49d8b55"
format_stamp: "Formatted at 2026-08-06 22:46:19 on dist-test-slave-zr1t"
server_key: "11c2897041803bac0a5cabe45d3dfa52"
server_key_iv: "ea695ea2fa404f165395987c1b5a2a07"
server_key_version: "encryptionkey@0"
I20260806 22:46:19.034554 22023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance:
uuid: "3cfc564e8d0e40f8914eb683e49d8b55"
format_stamp: "Formatted at 2026-08-06 22:46:19 on dist-test-slave-zr1t"
server_key: "11c2897041803bac0a5cabe45d3dfa52"
server_key_iv: "ea695ea2fa404f165395987c1b5a2a07"
server_key_version: "encryptionkey@0"
I20260806 22:46:19.039575 22023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:46:19.043644 22047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:19.045491 22023 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 22:46:19.045701 22023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/wal
uuid: "3cfc564e8d0e40f8914eb683e49d8b55"
format_stamp: "Formatted at 2026-08-06 22:46:19 on dist-test-slave-zr1t"
server_key: "11c2897041803bac0a5cabe45d3dfa52"
server_key_iv: "ea695ea2fa404f165395987c1b5a2a07"
server_key_version: "encryptionkey@0"
I20260806 22:46:19.045943 22023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:19.076812 22023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:19.077749 22023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:19.078071 22023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:19.079082 22023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 22:46:19.080574 22023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 22:46:19.080662 22023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:19.080821 22023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 22:46:19.080912 22023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:19.115401 22023 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.131:37855
I20260806 22:46:19.115568 22215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.131:37855 every 8 connection(s)
I20260806 22:46:19.116820 22023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/data/info.pb
I20260806 22:46:19.123576 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 22023
I20260806 22:46:19.123772 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/wal/instance
I20260806 22:46:19.124312 23954 external_mini_cluster.cc:1428] Setting key 3be8a35a6baa1186207681ce7717d078
I20260806 22:46:19.130060 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.119360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.131:55949 (local address 127.23.100.190:41399)
0806 22:46:19.119548 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:19.119552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:19.121085 (+ 1533us) server_negotiation.cc:407] Connection header received
0806 22:46:19.123050 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:19.123055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:19.123124 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:19.123211 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:19.124612 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.125707 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.127630 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.128039 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.128133 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.129373 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:19.129377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:19.129380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:19.129396 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:19.129423 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:19.129434 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:19.129503 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:19.129753 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 22:46:19.129826 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 22:46:19.131026 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.119931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:41399 (local address 127.23.100.131:55949)
0806 22:46:19.120895 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:19.120939 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:19.122804 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:19.123490 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:19.123502 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:19.123626 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:46:19.124439 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.124464 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.125877 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.125883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.127499 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.127512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.128219 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.128223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.128411 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.129096 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 22:46:19.129099 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:46:19.129137 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 22:46:19.129216 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:46:19.129559 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:46:19.129569 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:46:19.129582 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 22:46:19.129724 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 22:46:19.130010 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":260,"threads_started":1}
I20260806 22:46:19.132292 22217 heartbeater.cc:347] Connected to a master server at 127.23.100.190:41399
I20260806 22:46:19.132597 22217 heartbeater.cc:464] Registering TS with master...
I20260806 22:46:19.133294 22217 heartbeater.cc:511] Master 127.23.100.190:41399 requested a full tablet report, sending...
I20260806 22:46:19.134637 21551 ts_manager.cc:194] Registered new tserver with Master: 3cfc564e8d0e40f8914eb683e49d8b55 (127.23.100.131:37855)
I20260806 22:46:19.135423 21551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.100.131:55949
I20260806 22:46:19.139477 23954 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 22:46:19.150314 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.143631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.23.100.190:41399)
0806 22:46:19.143859 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:19.143867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:19.144310 (+ 443us) server_negotiation.cc:407] Connection header received
0806 22:46:19.144525 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:19.144530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:19.144603 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:19.144708 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:19.145220 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.146285 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.147694 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.148289 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.148442 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.149548 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:19.149554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:19.149560 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:19.149582 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:19.149621 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:19.149636 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:19.149719 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:19.149861 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 22:46:19.149959 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 22:46:19.154146 21551 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 22:46:19.161067 21551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 22:46:19.174829 21837 heartbeater.cc:503] Master 127.23.100.190:41399 was elected leader, sending a full tablet report...
I20260806 22:46:19.186664 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.175195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45097 (local address 127.0.0.1:42750)
0806 22:46:19.176746 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:19.176769 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:19.176922 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:19.179264 (+ 2342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:19.179271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:19.179295 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:19.179751 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.179765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.181849 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.181855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.184146 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.184157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.185575 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.185582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.185778 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.185829 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 22:46:19.185997 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 22:46:19.186207 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1381,"mutex_wait_us":3,"thread_start_us":1914,"threads_started":1}
I20260806 22:46:19.186865 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.173377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:37855 (local address 127.0.0.1:42776)
0806 22:46:19.174451 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:19.174515 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:19.174695 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:19.178984 (+ 4289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:19.178989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:19.179012 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:19.179476 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.179489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.181490 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.181496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.184121 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.184138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.186012 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.186018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.186198 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.186227 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 22:46:19.186312 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 22:46:19.186440 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":23,"thread_start_us":2307,"threads_started":1}
I20260806 22:46:19.187534 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.175729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.23.100.129:45097)
0806 22:46:19.178054 (+ 2325us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:19.178061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:19.178090 (+ 29us) server_negotiation.cc:407] Connection header received
0806 22:46:19.178194 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:19.178201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:19.178406 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:19.178620 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:19.179986 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.181694 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.184396 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.185328 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.185574 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.185810 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:19.186334 (+ 524us) server_negotiation.cc:299] Negotiation successful
0806 22:46:19.186516 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2106,"thread_start_us":1957,"threads_started":1}
I20260806 22:46:19.187816 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.173954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.23.100.131:37855)
0806 22:46:19.178384 (+ 4430us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:19.178391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:19.178424 (+ 33us) server_negotiation.cc:407] Connection header received
0806 22:46:19.178521 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:19.178528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:19.178747 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:19.179685 (+ 938us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:19.179990 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.181341 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.184360 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.185790 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.186012 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.186206 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:19.187282 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0806 22:46:19.187463 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4234,"thread_start_us":4069,"threads_started":1}
I20260806 22:46:19.187873 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.173415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.130:45833 (local address 127.0.0.1:39506)
0806 22:46:19.175267 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:19.175287 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:19.175399 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:19.178333 (+ 2934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:19.178341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:19.178379 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:19.178856 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.178871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.180948 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.180954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.183775 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.183788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.185953 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.185960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.186174 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.186210 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 22:46:19.187361 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0806 22:46:19.187518 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1731,"mutex_wait_us":176,"thread_start_us":775,"threads_started":1}
I20260806 22:46:19.188625 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.174050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.23.100.130:45833)
0806 22:46:19.177619 (+ 3569us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:19.177633 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:19.177673 (+ 40us) server_negotiation.cc:407] Connection header received
0806 22:46:19.177783 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:19.177790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:19.178102 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:19.178976 (+ 874us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:19.179230 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.180804 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.183973 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.185389 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.185590 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.185811 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:19.188110 (+ 2299us) server_negotiation.cc:299] Negotiation successful
0806 22:46:19.188275 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3240,"spinlock_wait_cycles":29440,"thread_start_us":3225,"threads_started":1}
I20260806 22:46:19.191582 21742 tablet_service.cc:1467] Processing CreateTablet for tablet 96ed8b0776c443eeb0187a0fb506112e (DEFAULT_TABLE table=test-table [id=49a106ce431c4faca0f315e62f0b6ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:19.191587 22120 tablet_service.cc:1467] Processing CreateTablet for tablet 96ed8b0776c443eeb0187a0fb506112e (DEFAULT_TABLE table=test-table [id=49a106ce431c4faca0f315e62f0b6ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:19.191570 21927 tablet_service.cc:1467] Processing CreateTablet for tablet 96ed8b0776c443eeb0187a0fb506112e (DEFAULT_TABLE table=test-table [id=49a106ce431c4faca0f315e62f0b6ec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 22:46:19.192984 21927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ed8b0776c443eeb0187a0fb506112e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:19.193050 21742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ed8b0776c443eeb0187a0fb506112e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:19.193048 22120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ed8b0776c443eeb0187a0fb506112e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:19.203894 22245 tablet_bootstrap.cc:492] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: Bootstrap starting.
I20260806 22:46:19.207540 22246 tablet_bootstrap.cc:492] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: Bootstrap starting.
I20260806 22:46:19.208613 22245 tablet_bootstrap.cc:654] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:19.209988 22245 log.cc:824] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:19.211179 22246 tablet_bootstrap.cc:654] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:19.211501 22247 tablet_bootstrap.cc:492] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: Bootstrap starting.
I20260806 22:46:19.212419 22246 log.cc:824] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:19.213428 22245 tablet_bootstrap.cc:492] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: No bootstrap required, opened a new log
I20260806 22:46:19.213771 22245 ts_tablet_manager.cc:1397] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260806 22:46:19.215113 22247 tablet_bootstrap.cc:654] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:19.215691 22246 tablet_bootstrap.cc:492] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: No bootstrap required, opened a new log
I20260806 22:46:19.215956 22246 ts_tablet_manager.cc:1397] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 22:46:19.216271 22247 log.cc:824] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:19.219295 22247 tablet_bootstrap.cc:492] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: No bootstrap required, opened a new log
I20260806 22:46:19.219550 22247 ts_tablet_manager.cc:1397] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 22:46:19.224678 22245 raft_consensus.cc:348] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 37855 } } peers { permanent_uuid: "bb8b8dcf038c4284a788b59946901ec1" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45833 } } peers { permanent_uuid: "9343103c26b8423889dae3d9c59585d9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45097 } }
I20260806 22:46:19.225167 22245 raft_consensus.cc:374] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:19.225266 22245 raft_consensus.cc:729] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8b8dcf038c4284a788b59946901ec1, State: Initialized, Role: FOLLOWER
I20260806 22:46:19.226069 22245 consensus_queue.cc:260] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 37855 } } peers { permanent_uuid: "bb8b8dcf038c4284a788b59946901ec1" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45833 } } peers { permanent_uuid: "9343103c26b8423889dae3d9c59585d9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45097 } }
I20260806 22:46:19.226691 22246 raft_consensus.cc:348] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 37855 } } peers { permanent_uuid: "bb8b8dcf038c4284a788b59946901ec1" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45833 } } peers { permanent_uuid: "9343103c26b8423889dae3d9c59585d9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45097 } }
I20260806 22:46:19.227116 22246 raft_consensus.cc:374] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:19.227221 22246 raft_consensus.cc:729] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9343103c26b8423889dae3d9c59585d9, State: Initialized, Role: FOLLOWER
I20260806 22:46:19.227228 22018 heartbeater.cc:503] Master 127.23.100.190:41399 was elected leader, sending a full tablet report...
I20260806 22:46:19.227977 22245 ts_tablet_manager.cc:1428] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260806 22:46:19.227914 22246 consensus_queue.cc:260] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 37855 } } peers { permanent_uuid: "bb8b8dcf038c4284a788b59946901ec1" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45833 } } peers { permanent_uuid: "9343103c26b8423889dae3d9c59585d9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45097 } }
I20260806 22:46:19.227924 22247 raft_consensus.cc:348] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 37855 } } peers { permanent_uuid: "bb8b8dcf038c4284a788b59946901ec1" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45833 } } peers { permanent_uuid: "9343103c26b8423889dae3d9c59585d9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45097 } }
I20260806 22:46:19.228329 22247 raft_consensus.cc:374] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:19.228417 22247 raft_consensus.cc:729] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfc564e8d0e40f8914eb683e49d8b55, State: Initialized, Role: FOLLOWER
I20260806 22:46:19.229763 22246 ts_tablet_manager.cc:1428] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260806 22:46:19.230350 22247 consensus_queue.cc:260] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 37855 } } peers { permanent_uuid: "bb8b8dcf038c4284a788b59946901ec1" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45833 } } peers { permanent_uuid: "9343103c26b8423889dae3d9c59585d9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45097 } }
I20260806 22:46:19.231460 22217 heartbeater.cc:503] Master 127.23.100.190:41399 was elected leader, sending a full tablet report...
I20260806 22:46:19.231983 22247 ts_tablet_manager.cc:1428] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260806 22:46:19.368113 22021 tablet.cc:2369] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:46:19.368537 22218 tablet.cc:2369] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 22:46:19.400415 21838 tablet.cc:2369] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:46:19.490048 22251 raft_consensus.cc:482] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 22:46:19.490348 22251 raft_consensus.cc:504] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 37855 } } peers { permanent_uuid: "bb8b8dcf038c4284a788b59946901ec1" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45833 } } peers { permanent_uuid: "9343103c26b8423889dae3d9c59585d9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45097 } }
I20260806 22:46:19.492411 22251 leader_election.cc:288] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cfc564e8d0e40f8914eb683e49d8b55 (127.23.100.131:37855), 9343103c26b8423889dae3d9c59585d9 (127.23.100.129:45097)
I20260806 22:46:19.499428 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.493537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:48397 (local address 127.23.100.129:45097)
0806 22:46:19.493756 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:19.493762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:19.493976 (+ 214us) server_negotiation.cc:407] Connection header received
0806 22:46:19.494193 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:19.494198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:19.494278 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:19.494376 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:19.495047 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.495725 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.497743 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.498522 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.498688 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.498759 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:19.499057 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 22:46:19.499179 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 22:46:19.499500 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.492885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.100.130:60899 (local address 127.23.100.131:37855)
0806 22:46:19.493140 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:19.493149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:19.493939 (+ 790us) server_negotiation.cc:407] Connection header received
0806 22:46:19.494176 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:19.494182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:19.494255 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:19.494355 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 22:46:19.494980 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.495709 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.497786 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.498587 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.498767 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.498833 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:19.499127 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 22:46:19.499248 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 22:46:19.499593 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.492911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.129:45097 (local address 127.23.100.130:48397)
0806 22:46:19.493869 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:19.493883 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:19.494058 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:19.494391 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:19.494395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:19.494418 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:19.494892 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.494901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.495860 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.495866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.497510 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.497571 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.498719 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.498726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.498888 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.498912 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 22:46:19.499017 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 22:46:19.499141 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":89,"spinlock_wait_cycles":26112,"thread_start_us":662,"threads_started":1}
I20260806 22:46:19.499656 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.492896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.131:37855 (local address 127.23.100.130:60899)
0806 22:46:19.493794 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0806 22:46:19.493819 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 22:46:19.494034 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:46:19.494378 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:46:19.494383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:46:19.494406 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 22:46:19.494817 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.494830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.495853 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.495859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.497566 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:46:19.497588 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.498782 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.498788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:46:19.498967 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.498990 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 22:46:19.499089 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 22:46:19.499208 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":51,"spinlock_wait_cycles":768,"thread_start_us":468,"threads_started":1}
I20260806 22:46:19.500664 22153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ed8b0776c443eeb0187a0fb506112e" candidate_uuid: "bb8b8dcf038c4284a788b59946901ec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" is_pre_election: true
I20260806 22:46:19.500737 21772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ed8b0776c443eeb0187a0fb506112e" candidate_uuid: "bb8b8dcf038c4284a788b59946901ec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9343103c26b8423889dae3d9c59585d9" is_pre_election: true
I20260806 22:46:19.501196 22153 raft_consensus.cc:2454] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb8b8dcf038c4284a788b59946901ec1 in term 0.
I20260806 22:46:19.501263 21772 raft_consensus.cc:2454] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb8b8dcf038c4284a788b59946901ec1 in term 0.
I20260806 22:46:19.501965 21869 leader_election.cc:302] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3cfc564e8d0e40f8914eb683e49d8b55, 9343103c26b8423889dae3d9c59585d9, bb8b8dcf038c4284a788b59946901ec1; no voters:
I20260806 22:46:19.502372 22251 raft_consensus.cc:2781] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 22:46:19.502518 22251 raft_consensus.cc:482] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 22:46:19.502629 22251 raft_consensus.cc:3037] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:19.504874 22251 raft_consensus.cc:504] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 37855 } } peers { permanent_uuid: "bb8b8dcf038c4284a788b59946901ec1" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45833 } } peers { permanent_uuid: "9343103c26b8423889dae3d9c59585d9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45097 } }
I20260806 22:46:19.505589 22251 leader_election.cc:288] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [CANDIDATE]: Term 1 election: Requested vote from peers 3cfc564e8d0e40f8914eb683e49d8b55 (127.23.100.131:37855), 9343103c26b8423889dae3d9c59585d9 (127.23.100.129:45097)
I20260806 22:46:19.506079 22153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ed8b0776c443eeb0187a0fb506112e" candidate_uuid: "bb8b8dcf038c4284a788b59946901ec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfc564e8d0e40f8914eb683e49d8b55"
I20260806 22:46:19.506081 21772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ed8b0776c443eeb0187a0fb506112e" candidate_uuid: "bb8b8dcf038c4284a788b59946901ec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9343103c26b8423889dae3d9c59585d9"
I20260806 22:46:19.506301 22153 raft_consensus.cc:3037] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:19.506304 21772 raft_consensus.cc:3037] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:19.508479 22153 raft_consensus.cc:2454] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8b8dcf038c4284a788b59946901ec1 in term 1.
I20260806 22:46:19.508500 21772 raft_consensus.cc:2454] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb8b8dcf038c4284a788b59946901ec1 in term 1.
I20260806 22:46:19.508894 21869 leader_election.cc:302] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cfc564e8d0e40f8914eb683e49d8b55, bb8b8dcf038c4284a788b59946901ec1; no voters:
I20260806 22:46:19.509194 22251 raft_consensus.cc:2781] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:19.509691 22251 raft_consensus.cc:686] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 1 LEADER]: Becoming Leader. State: Replica: bb8b8dcf038c4284a788b59946901ec1, State: Running, Role: LEADER
I20260806 22:46:19.510154 22251 consensus_queue.cc:237] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 37855 } } peers { permanent_uuid: "bb8b8dcf038c4284a788b59946901ec1" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45833 } } peers { permanent_uuid: "9343103c26b8423889dae3d9c59585d9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45097 } }
I20260806 22:46:19.515069 21551 catalog_manager.cc:5495] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb8b8dcf038c4284a788b59946901ec1 (127.23.100.130). New cstate: current_term: 1 leader_uuid: "bb8b8dcf038c4284a788b59946901ec1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 37855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb8b8dcf038c4284a788b59946901ec1" member_type: VOTER last_known_addr { host: "127.23.100.130" port: 45833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9343103c26b8423889dae3d9c59585d9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45097 } health_report { overall_health: UNKNOWN } } }
I20260806 22:46:19.552214 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.546838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.23.100.130:45833)
0806 22:46:19.547107 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:19.547112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:19.547132 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:46:19.547315 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:19.547318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:19.547406 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:19.547533 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:46:19.547862 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.548575 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.549983 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.550419 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.550575 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.551081 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:46:19.551594 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:46:19.551756 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:19.551817 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 22:46:19.551899 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 22:46:19.560564 21772 raft_consensus.cc:1261] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9 [term 1 FOLLOWER]: Refusing update from remote peer bb8b8dcf038c4284a788b59946901ec1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:19.560616 22153 raft_consensus.cc:1261] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55 [term 1 FOLLOWER]: Refusing update from remote peer bb8b8dcf038c4284a788b59946901ec1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:46:19.561623 22259 consensus_queue.cc:1035] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cfc564e8d0e40f8914eb683e49d8b55" member_type: VOTER last_known_addr { host: "127.23.100.131" port: 37855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:46:19.562037 22251 consensus_queue.cc:1035] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9343103c26b8423889dae3d9c59585d9" member_type: VOTER last_known_addr { host: "127.23.100.129" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:46:19.591173 22267 mvcc.cc:205] Tried to move back new op lower bound from 7315686930669502464 to 7315686930477490176. Current Snapshot: MvccSnapshot[applied={T|T < 7315686930669502464}]
I20260806 22:46:19.604635 21549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:19.605010 21549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 22:46:19.610656 21549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffb2af92185244458fdac3d8adf58f5c: Sending DeleteTablet for 3 replicas of tablet 96ed8b0776c443eeb0187a0fb506112e
I20260806 22:46:19.612124 22120 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ed8b0776c443eeb0187a0fb506112e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:19 UTC) from {username='slave'} at 127.0.0.1:42776
I20260806 22:46:19.614337 21927 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ed8b0776c443eeb0187a0fb506112e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:19 UTC) from {username='slave'} at 127.0.0.1:39506
I20260806 22:46:19.622798 21742 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ed8b0776c443eeb0187a0fb506112e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 22:46:19 UTC) from {username='slave'} at 127.0.0.1:42750
I20260806 22:46:19.624763 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 21643
I20260806 22:46:19.642747 22280 tablet_replica.cc:330] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: stopping tablet replica
I20260806 22:46:19.643683 22280 raft_consensus.cc:2229] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 22:46:19.644419 22278 tablet_replica.cc:330] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: stopping tablet replica
I20260806 22:46:19.645257 22280 raft_consensus.cc:2258] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:19.647230 22281 tablet_replica.cc:330] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: stopping tablet replica
I20260806 22:46:19.647746 22278 raft_consensus.cc:2229] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:19.648289 22278 raft_consensus.cc:2258] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:19.649044 22281 raft_consensus.cc:2229] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 22:46:19.649977 22281 raft_consensus.cc:2258] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 22:46:19.651295 22280 ts_tablet_manager.cc:1905] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:19.653169 22278 ts_tablet_manager.cc:1905] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:19.659284 22280 ts_tablet_manager.cc:1918] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:19.659471 22280 log.cc:1196] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-1/wal/wals/96ed8b0776c443eeb0187a0fb506112e
I20260806 22:46:19.659554 22281 ts_tablet_manager.cc:1905] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 22:46:19.660091 22280 ts_tablet_manager.cc:1939] T 96ed8b0776c443eeb0187a0fb506112e P bb8b8dcf038c4284a788b59946901ec1: Deleting consensus metadata
I20260806 22:46:19.662300 21529 catalog_manager.cc:4841] TS bb8b8dcf038c4284a788b59946901ec1 (127.23.100.130:45833): tablet 96ed8b0776c443eeb0187a0fb506112e (table test-table [id=49a106ce431c4faca0f315e62f0b6ec0]) successfully deleted
I20260806 22:46:19.665091 22281 ts_tablet_manager.cc:1918] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:19.665273 22281 log.cc:1196] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-0/wal/wals/96ed8b0776c443eeb0187a0fb506112e
I20260806 22:46:19.665867 22281 ts_tablet_manager.cc:1939] T 96ed8b0776c443eeb0187a0fb506112e P 9343103c26b8423889dae3d9c59585d9: Deleting consensus metadata
I20260806 22:46:19.667421 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.627363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.23.100.129:45097)
0806 22:46:19.627603 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:19.627613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:19.637842 (+ 10229us) server_negotiation.cc:407] Connection header received
0806 22:46:19.643587 (+ 5745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:19.643596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:19.643734 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:19.643826 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:19.645122 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.646339 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.648357 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.648998 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.649109 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.665508 (+ 16399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:19.665546 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:19.665563 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:19.666359 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:19.666428 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:19.666455 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:19.666548 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:19.666946 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 22:46:19.667070 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:46:19.667691 21529 catalog_manager.cc:4841] TS 9343103c26b8423889dae3d9c59585d9 (127.23.100.129:45097): tablet 96ed8b0776c443eeb0187a0fb506112e (table test-table [id=49a106ce431c4faca0f315e62f0b6ec0]) successfully deleted
I20260806 22:46:19.668738 21827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:19.671900 22278 ts_tablet_manager.cc:1918] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 22:46:19.672226 22278 log.cc:1196] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: Deleting WAL directory at /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786056046488940-23954-0/minicluster-data/ts-2/wal/wals/96ed8b0776c443eeb0187a0fb506112e
I20260806 22:46:19.672960 22278 ts_tablet_manager.cc:1939] T 96ed8b0776c443eeb0187a0fb506112e P 3cfc564e8d0e40f8914eb683e49d8b55: Deleting consensus metadata
I20260806 22:46:19.674582 21531 catalog_manager.cc:4841] TS 3cfc564e8d0e40f8914eb683e49d8b55 (127.23.100.131:37855): tablet 96ed8b0776c443eeb0187a0fb506112e (table test-table [id=49a106ce431c4faca0f315e62f0b6ec0]) successfully deleted
I20260806 22:46:19.791344 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 21643
I20260806 22:46:19.823734 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 21843
I20260806 22:46:19.831713 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.824543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.23.100.130:45833)
0806 22:46:19.824745 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:19.824751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:19.824772 (+ 21us) server_negotiation.cc:407] Connection header received
0806 22:46:19.824923 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:19.824929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:19.825056 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:19.825181 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:19.825562 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.826782 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.828074 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.828780 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.828916 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.829990 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:19.830001 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:19.830020 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:19.830873 (+ 853us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:19.830952 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:19.830983 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:19.831094 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:19.831223 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 22:46:19.831345 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 22:46:19.832495 22012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:19.949071 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 21843
I20260806 22:46:19.973397 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 22023
I20260806 22:46:19.980489 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:19.974204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.23.100.131:37855)
0806 22:46:19.974400 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:19.974406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:19.974511 (+ 105us) server_negotiation.cc:407] Connection header received
0806 22:46:19.974738 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:19.974749 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:19.974837 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:19.974957 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:19.975376 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.976102 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.977376 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:19.977875 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:19.978023 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:19.979093 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:19.979103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:19.979118 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:19.979694 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:19.979744 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:19.979765 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:19.979906 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:19.980036 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 22:46:19.980152 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 22:46:19.981177 22211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:20.071748 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 22023
I20260806 22:46:20.096133 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 21505
I20260806 22:46:20.097271 21602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:20.207571 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 21505
2026-08-06T22:46:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3197 ms)
[----------] 6 tests from EncryptionPolicyTest (24210 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:20 dist-test-slave-zr1t krb5kdc[22289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:20 dist-test-slave-zr1t krb5kdc[22289](info): set up 2 sockets
Aug 06 22:46:20 dist-test-slave-zr1t krb5kdc[22289](info): commencing operation
krb5kdc: starting...
W20260806 22:46:21.546574 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:21 dist-test-slave-zr1t krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:21.884641 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:22.235663 22307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:22.235963 22307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:22.236059 22307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:22.243927 22307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:22.244055 22307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:22.244102 22307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:22.244167 22307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:22.244211 22307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:22.254416 22307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:22.255795 22307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:22.257141 22307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:22.269021 22319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:22.269102 22317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:22.269062 22321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:22.269910 22307 server_base.cc:1034] running on GCE node
I20260806 22:46:22.270738 22307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:22.272198 22307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:22.273430 22307 hybrid_clock.cc:648] HybridClock initialized: now 1786056382273370 us; error 54 us; skew 500 ppm
I20260806 22:46:22.273865 22307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:22 dist-test-slave-zr1t krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:22.281940 22307 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:22.283490 22307 webserver.cc:460] Webserver started at http://127.0.0.1:39093/ using document root <none> and password file <none>
I20260806 22:46:22.284210 22307 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:22.284365 22307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:22.284693 22307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:22.287829 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "d34c2db6d3c54d5f9ec8bc85c6ee88c7"
format_stamp: "Formatted at 2026-08-06 22:46:22 on dist-test-slave-zr1t"
server_key: "8b897712c0819b91be9df4a6e9c28905"
server_key_iv: "64c8d0b1292ca2eb111dbb72d648dbd7"
server_key_version: "encryptionkey@0"
I20260806 22:46:22.288652 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "d34c2db6d3c54d5f9ec8bc85c6ee88c7"
format_stamp: "Formatted at 2026-08-06 22:46:22 on dist-test-slave-zr1t"
server_key: "8b897712c0819b91be9df4a6e9c28905"
server_key_iv: "64c8d0b1292ca2eb111dbb72d648dbd7"
server_key_version: "encryptionkey@0"
I20260806 22:46:22.293982 22307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 22:46:22.297442 22331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:22.298899 22307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:46:22.299125 22307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "d34c2db6d3c54d5f9ec8bc85c6ee88c7"
format_stamp: "Formatted at 2026-08-06 22:46:22 on dist-test-slave-zr1t"
server_key: "8b897712c0819b91be9df4a6e9c28905"
server_key_iv: "64c8d0b1292ca2eb111dbb72d648dbd7"
server_key_version: "encryptionkey@0"
I20260806 22:46:22.299340 22307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:22.316128 22307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:22.320619 22307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:22.320916 22307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:22.340245 22307 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33549
I20260806 22:46:22.340278 22407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33549 every 8 connection(s)
I20260806 22:46:22.341665 22307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:22.345651 22408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:22.348886 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 22307
I20260806 22:46:22.349134 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:22.350000 23954 external_mini_cluster.cc:1428] Setting key a1a35d38eaabb1bb94b7de8cc3e8a32f
I20260806 22:46:22.358711 22408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7: Bootstrap starting.
I20260806 22:46:22.362568 22408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:22.363903 22408 log.cc:824] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7: Log is configured to *not* fsync() on all Append() calls
Aug 06 22:46:22 dist-test-slave-zr1t krb5kdc[22289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 22:46:22.367141 22408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7: No bootstrap required, opened a new log
I20260806 22:46:22.375365 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:22.351982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41299 (local address 127.0.0.1:33549)
0806 22:46:22.352780 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:22.352792 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:22.352834 (+ 42us) server_negotiation.cc:407] Connection header received
0806 22:46:22.354576 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:22.354598 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:22.355018 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:22.355334 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:22.357197 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:22.358666 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:22.360469 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:22.361127 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:22.361255 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:22.366515 (+ 5260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:22.366531 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:22.366552 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:22.366594 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:22.370692 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:22.371499 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:22.371508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:22.371516 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:22.371730 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:22.372173 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:22.372178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:22.372182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:22.372871 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:22.373214 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:22.373619 (+ 405us) server_negotiation.cc:299] Negotiation successful
0806 22:46:22.373987 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":388,"threads_started":1}
I20260806 22:46:22.377694 22408 raft_consensus.cc:348] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34c2db6d3c54d5f9ec8bc85c6ee88c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } }
I20260806 22:46:22.378069 22408 raft_consensus.cc:374] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:22.378189 22408 raft_consensus.cc:729] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34c2db6d3c54d5f9ec8bc85c6ee88c7, State: Initialized, Role: FOLLOWER
I20260806 22:46:22.378755 22408 consensus_queue.cc:260] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34c2db6d3c54d5f9ec8bc85c6ee88c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } }
I20260806 22:46:22.379011 22408 raft_consensus.cc:388] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:22.379184 22408 raft_consensus.cc:482] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:22.379364 22408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:22.379732 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:22.382726 22408 raft_consensus.cc:504] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34c2db6d3c54d5f9ec8bc85c6ee88c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } }
I20260806 22:46:22.383433 22408 leader_election.cc:302] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d34c2db6d3c54d5f9ec8bc85c6ee88c7; no voters:
I20260806 22:46:22.384243 22408 leader_election.cc:288] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:22.384397 22417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:22.387290 22408 sys_catalog.cc:564] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:22.387538 22417 raft_consensus.cc:686] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [term 1 LEADER]: Becoming Leader. State: Replica: d34c2db6d3c54d5f9ec8bc85c6ee88c7, State: Running, Role: LEADER
I20260806 22:46:22.388185 22417 consensus_queue.cc:237] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34c2db6d3c54d5f9ec8bc85c6ee88c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } }
I20260806 22:46:22.391973 22424 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d34c2db6d3c54d5f9ec8bc85c6ee88c7. Latest consensus state: current_term: 1 leader_uuid: "d34c2db6d3c54d5f9ec8bc85c6ee88c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34c2db6d3c54d5f9ec8bc85c6ee88c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } } }
I20260806 22:46:22.391985 22423 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d34c2db6d3c54d5f9ec8bc85c6ee88c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34c2db6d3c54d5f9ec8bc85c6ee88c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33549 } } }
I20260806 22:46:22.392493 22423 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:22.392493 22424 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:22.397351 22438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:22.403380 22438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:22.408387 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:22.386135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51717 (local address 127.0.0.1:33549)
0806 22:46:22.386319 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:22.386326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:22.387010 (+ 684us) server_negotiation.cc:407] Connection header received
0806 22:46:22.387301 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:22.387309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:22.387401 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:22.387486 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:22.389992 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:22.391539 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:22.393798 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:22.394759 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:22.394909 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:22.399186 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:22.399191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:22.399194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:22.399227 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:22.403300 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:22.404652 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:22.404665 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:22.404671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:22.404874 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:22.405698 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:22.405707 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:22.405712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:22.406272 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:22.406639 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:22.407546 (+ 907us) server_negotiation.cc:299] Negotiation successful
0806 22:46:22.407818 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 22:46:22.419606 22438 catalog_manager.cc:1347] Generated new cluster ID: 66fc7932bd8e49c199ff55adf85040db
I20260806 22:46:22.419720 22438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:22.463078 22438 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:22.464416 22438 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:22.473560 22438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d34c2db6d3c54d5f9ec8bc85c6ee88c7: Generated new TSK 0
I20260806 22:46:22.474462 22438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:22.485194 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 22307
I20260806 22:46:22.486716 22402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:22.581370 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 22307
W20260806 22:46:22.606380 22287 connection.cc:535] client connection to 127.0.0.1:33549 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33549): Connection reset by peer (error 104)
2026-08-06T22:46:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:22 dist-test-slave-zr1t krb5kdc[22456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:22 dist-test-slave-zr1t krb5kdc[22456](info): set up 2 sockets
Aug 06 22:46:22 dist-test-slave-zr1t krb5kdc[22456](info): commencing operation
krb5kdc: starting...
W20260806 22:46:23.910796 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:24 dist-test-slave-zr1t krb5kdc[22456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:24.252789 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:24.602138 22476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:24.602480 22476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:24.602649 22476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:24.610072 22476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:24.610189 22476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:24.610301 22476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:24.610406 22476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:24.610491 22476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:24.620450 22476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43507
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:24.621825 22476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:24.623169 22476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:24.635658 22485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:24.635617 22488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:24.636360 22484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:24.637271 22476 server_base.cc:1034] running on GCE node
I20260806 22:46:24.638248 22476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:24.639786 22476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:24.641033 22476 hybrid_clock.cc:648] HybridClock initialized: now 1786056384641035 us; error 116 us; skew 500 ppm
I20260806 22:46:24.641407 22476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:24 dist-test-slave-zr1t krb5kdc[22456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:24.644759 22476 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:24.646409 22476 webserver.cc:460] Webserver started at http://127.0.0.1:42401/ using document root <none> and password file <none>
I20260806 22:46:24.647123 22476 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:24.647280 22476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:24.647610 22476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:24.650838 22476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "f23a288a3022436c996a8b46b021dfc5"
format_stamp: "Formatted at 2026-08-06 22:46:24 on dist-test-slave-zr1t"
server_key: "46ffe129433db9dc7c4f3aeb78b27add"
server_key_iv: "651e3217e091b74667dbd4cbf2e6e59d"
server_key_version: "encryptionkey@0"
I20260806 22:46:24.651664 22476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "f23a288a3022436c996a8b46b021dfc5"
format_stamp: "Formatted at 2026-08-06 22:46:24 on dist-test-slave-zr1t"
server_key: "46ffe129433db9dc7c4f3aeb78b27add"
server_key_iv: "651e3217e091b74667dbd4cbf2e6e59d"
server_key_version: "encryptionkey@0"
I20260806 22:46:24.657639 22476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 22:46:24.661324 22498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:24.662786 22476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:46:24.662974 22476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "f23a288a3022436c996a8b46b021dfc5"
format_stamp: "Formatted at 2026-08-06 22:46:24 on dist-test-slave-zr1t"
server_key: "46ffe129433db9dc7c4f3aeb78b27add"
server_key_iv: "651e3217e091b74667dbd4cbf2e6e59d"
server_key_version: "encryptionkey@0"
I20260806 22:46:24.663183 22476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:24.684751 22476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:24.689342 22476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:24.689711 22476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:24.709697 22476 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43507
I20260806 22:46:24.709753 22576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43507 every 8 connection(s)
I20260806 22:46:24.711102 22476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:24.715003 22577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:24.717510 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 22476
I20260806 22:46:24.717741 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:24.718284 23954 external_mini_cluster.cc:1428] Setting key 6cd5cb03691793f6566510c1529850f7
I20260806 22:46:24.730350 22577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5: Bootstrap starting.
I20260806 22:46:24.732577 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:24.720039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49821 (local address 127.0.0.1:43507)
0806 22:46:24.720947 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:24.720959 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:24.721002 (+ 43us) server_negotiation.cc:407] Connection header received
0806 22:46:24.722687 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:24.722710 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:24.723119 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:24.723442 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:24.724060 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:24.725684 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:24.727729 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:24.728420 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:24.728560 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:24.730017 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:24.730027 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:24.730041 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:24.730663 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:24.730742 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:24.730767 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:24.730916 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:24.731399 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 22:46:24.731599 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":421,"threads_started":1}
I20260806 22:46:24.734122 22577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:24.735296 22577 log.cc:824] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:24.736925 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:24.738706 22577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5: No bootstrap required, opened a new log
I20260806 22:46:24.747874 22577 raft_consensus.cc:348] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23a288a3022436c996a8b46b021dfc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } }
I20260806 22:46:24.748417 22577 raft_consensus.cc:374] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:24.748523 22577 raft_consensus.cc:729] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23a288a3022436c996a8b46b021dfc5, State: Initialized, Role: FOLLOWER
I20260806 22:46:24.749073 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:24.741861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56489 (local address 127.0.0.1:43507)
0806 22:46:24.742068 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:24.742074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:24.742480 (+ 406us) server_negotiation.cc:407] Connection header received
0806 22:46:24.742635 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:24.742646 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:24.742726 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:24.742858 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:24.743429 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:24.744605 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:24.746085 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:24.746809 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:24.746951 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:24.748124 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:24.748131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:24.748135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:24.748157 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:24.748230 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:24.748253 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:24.748393 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:24.748691 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 22:46:24.748746 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 22:46:24.749311 22577 consensus_queue.cc:260] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23a288a3022436c996a8b46b021dfc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } }
I20260806 22:46:24.749632 22577 raft_consensus.cc:388] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:24.749799 22577 raft_consensus.cc:482] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:24.749993 22577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:24.752321 22577 raft_consensus.cc:504] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23a288a3022436c996a8b46b021dfc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } }
I20260806 22:46:24.752892 22577 leader_election.cc:302] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f23a288a3022436c996a8b46b021dfc5; no voters:
I20260806 22:46:24.753504 22577 leader_election.cc:288] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:24.753711 22595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:24.754482 22595 raft_consensus.cc:686] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [term 1 LEADER]: Becoming Leader. State: Replica: f23a288a3022436c996a8b46b021dfc5, State: Running, Role: LEADER
I20260806 22:46:24.755048 22595 consensus_queue.cc:237] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23a288a3022436c996a8b46b021dfc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } }
I20260806 22:46:24.755872 22577 sys_catalog.cc:564] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:24.758379 22597 sys_catalog.cc:455] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f23a288a3022436c996a8b46b021dfc5. Latest consensus state: current_term: 1 leader_uuid: "f23a288a3022436c996a8b46b021dfc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23a288a3022436c996a8b46b021dfc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } } }
I20260806 22:46:24.758883 22597 sys_catalog.cc:458] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:24.761770 22596 sys_catalog.cc:455] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f23a288a3022436c996a8b46b021dfc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23a288a3022436c996a8b46b021dfc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } } }
I20260806 22:46:24.762215 22596 sys_catalog.cc:458] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:24.762988 22606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:24.768589 22606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:24.778976 22606 catalog_manager.cc:1347] Generated new cluster ID: d47e3f71ed5247e8b1c6fd18ee36307f
I20260806 22:46:24.779181 22606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:24.788394 22606 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:24.790249 22606 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:24.801193 22606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f23a288a3022436c996a8b46b021dfc5: Generated new TSK 0
I20260806 22:46:24.802100 22606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:24.815096 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 22476
W20260806 22:46:24.816306 22566 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49821
W20260806 22:46:24.817200 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 22476: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:46:24.817466 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 22476
W20260806 22:46:24.841140 22454 connection.cc:535] client connection to 127.0.0.1:43507 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43507): Connection reset by peer (error 104)
2026-08-06T22:46:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2234 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:24 dist-test-slave-zr1t krb5kdc[22624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:24 dist-test-slave-zr1t krb5kdc[22624](info): set up 2 sockets
Aug 06 22:46:24 dist-test-slave-zr1t krb5kdc[22624](info): commencing operation
krb5kdc: starting...
W20260806 22:46:26.167116 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.306s user 0.001s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:26 dist-test-slave-zr1t krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:26.503257 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:26.853497 22639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:26.853904 22639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:26.854100 22639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:26.861622 22639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:26.861757 22639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:26.861863 22639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:26.861972 22639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:26.862056 22639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:26.872200 22639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:26.873631 22639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:26.875298 22639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:26.887477 22648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:26.887526 22644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:26.887529 22646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:26.889195 22639 server_base.cc:1034] running on GCE node
I20260806 22:46:26.890118 22639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:26.891657 22639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:26.892887 22639 hybrid_clock.cc:648] HybridClock initialized: now 1786056386892827 us; error 39 us; skew 500 ppm
I20260806 22:46:26.893265 22639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:26 dist-test-slave-zr1t krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:26.896783 22639 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:26.898451 22639 webserver.cc:460] Webserver started at http://127.0.0.1:33173/ using document root <none> and password file <none>
I20260806 22:46:26.899170 22639 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:26.899324 22639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:26.899647 22639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:26.902767 22639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "dbdf2c8ed6dc4bfa8fa1c9bbf10af966"
format_stamp: "Formatted at 2026-08-06 22:46:26 on dist-test-slave-zr1t"
server_key: "c861f3cca9e63337365e252f30363f20"
server_key_iv: "8bd87472c162a803b3bdd436933162f8"
server_key_version: "encryptionkey@0"
I20260806 22:46:26.903565 22639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "dbdf2c8ed6dc4bfa8fa1c9bbf10af966"
format_stamp: "Formatted at 2026-08-06 22:46:26 on dist-test-slave-zr1t"
server_key: "c861f3cca9e63337365e252f30363f20"
server_key_iv: "8bd87472c162a803b3bdd436933162f8"
server_key_version: "encryptionkey@0"
I20260806 22:46:26.908859 22639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:46:26.912535 22659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:26.913940 22639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 22:46:26.914127 22639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "dbdf2c8ed6dc4bfa8fa1c9bbf10af966"
format_stamp: "Formatted at 2026-08-06 22:46:26 on dist-test-slave-zr1t"
server_key: "c861f3cca9e63337365e252f30363f20"
server_key_iv: "8bd87472c162a803b3bdd436933162f8"
server_key_version: "encryptionkey@0"
I20260806 22:46:26.914338 22639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:26.920964 22639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:26.925498 22639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:26.925890 22639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:26.945271 22639 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40667
I20260806 22:46:26.945319 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40667 every 8 connection(s)
I20260806 22:46:26.946805 22639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:26.947870 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 22639
I20260806 22:46:26.948062 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:26.948567 23954 external_mini_cluster.cc:1428] Setting key e24bd9e683cc191d1c740f051a1c150a
I20260806 22:46:26.951661 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:26.958276 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:26.950386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33943 (local address 127.0.0.1:40667)
0806 22:46:26.951297 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:26.951314 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:26.951621 (+ 307us) server_negotiation.cc:407] Connection header received
0806 22:46:26.954237 (+ 2616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:26.954266 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:26.954729 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:26.955165 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:26.956184 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:26.956193 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:26.956209 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:26.956227 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:26.956326 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:26.956373 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:26.956593 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:26.956738 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 22:46:26.956939 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":389,"threads_started":1}
I20260806 22:46:26.962514 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:26.964884 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966: Bootstrap starting.
I20260806 22:46:26.968497 22737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:26.970187 22737 log.cc:824] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:26.972149 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:26.968822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58355 (local address 127.0.0.1:40667)
0806 22:46:26.969052 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:26.969058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:26.970008 (+ 950us) server_negotiation.cc:407] Connection header received
0806 22:46:26.970226 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:26.970230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:26.970305 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:26.970409 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:26.971458 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:26.971462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:26.971466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:26.971481 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:26.971521 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:26.971536 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:26.971685 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:26.971802 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 22:46:26.971863 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 22:46:26.973619 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966: No bootstrap required, opened a new log
I20260806 22:46:26.981498 22737 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdf2c8ed6dc4bfa8fa1c9bbf10af966" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } }
I20260806 22:46:26.981992 22737 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:26.982069 22737 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdf2c8ed6dc4bfa8fa1c9bbf10af966, State: Initialized, Role: FOLLOWER
I20260806 22:46:26.982756 22737 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdf2c8ed6dc4bfa8fa1c9bbf10af966" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } }
I20260806 22:46:26.983037 22737 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:26.983206 22737 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:26.983387 22737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:26.985677 22737 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdf2c8ed6dc4bfa8fa1c9bbf10af966" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } }
I20260806 22:46:26.986191 22737 leader_election.cc:302] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbdf2c8ed6dc4bfa8fa1c9bbf10af966; no voters:
I20260806 22:46:26.986814 22737 leader_election.cc:288] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:26.987041 22748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:26.987952 22748 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [term 1 LEADER]: Becoming Leader. State: Replica: dbdf2c8ed6dc4bfa8fa1c9bbf10af966, State: Running, Role: LEADER
I20260806 22:46:26.988600 22748 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdf2c8ed6dc4bfa8fa1c9bbf10af966" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } }
I20260806 22:46:26.989059 22737 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:26.992030 22750 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbdf2c8ed6dc4bfa8fa1c9bbf10af966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdf2c8ed6dc4bfa8fa1c9bbf10af966" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } } }
I20260806 22:46:26.992149 22752 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbdf2c8ed6dc4bfa8fa1c9bbf10af966. Latest consensus state: current_term: 1 leader_uuid: "dbdf2c8ed6dc4bfa8fa1c9bbf10af966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdf2c8ed6dc4bfa8fa1c9bbf10af966" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } } }
I20260806 22:46:26.992884 22750 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:26.992939 22752 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:26.993959 22760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:27.000589 22760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:27.010210 22760 catalog_manager.cc:1347] Generated new cluster ID: 50b41bdcee934d549a8f03ca0805cfa4
I20260806 22:46:27.010353 22760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:27.029794 22760 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:27.030300 22760 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:27.046626 22760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbdf2c8ed6dc4bfa8fa1c9bbf10af966: Generated new TSK 0
I20260806 22:46:27.048089 22760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:27.055442 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 22639
W20260806 22:46:27.057731 22728 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33943
W20260806 22:46:27.058526 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 22639: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:46:27.058697 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 22639
2026-08-06T22:46:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:27 dist-test-slave-zr1t krb5kdc[22778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:27 dist-test-slave-zr1t krb5kdc[22778](info): set up 2 sockets
Aug 06 22:46:27 dist-test-slave-zr1t krb5kdc[22778](info): commencing operation
krb5kdc: starting...
W20260806 22:46:28.404284 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:28 dist-test-slave-zr1t krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:28.742224 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:29.091773 22796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:29.092057 22796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:29.092151 22796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:29.099334 22796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:29.099444 22796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:29.099498 22796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:29.099560 22796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:29.099604 22796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:29.109352 22796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:29.110692 22796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:29.111980 22796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:29.123796 22805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:29.123916 22802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:29.124045 22807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:29.124593 22796 server_base.cc:1034] running on GCE node
I20260806 22:46:29.125337 22796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:29.126812 22796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:29.128051 22796 hybrid_clock.cc:648] HybridClock initialized: now 1786056389128003 us; error 44 us; skew 500 ppm
I20260806 22:46:29.128404 22796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:29 dist-test-slave-zr1t krb5kdc[22778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:29.131605 22796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:29.133073 22796 webserver.cc:460] Webserver started at http://127.0.0.1:41539/ using document root <none> and password file <none>
I20260806 22:46:29.133773 22796 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:29.133881 22796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:29.134128 22796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:29.137190 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "d889534999c34ff6af9802834ab5715f"
format_stamp: "Formatted at 2026-08-06 22:46:29 on dist-test-slave-zr1t"
server_key: "4c234dd3a801ff3d234dfd6b5756a2d8"
server_key_iv: "4eceb7ce90ee98091919559da3f6c5c1"
server_key_version: "encryptionkey@0"
I20260806 22:46:29.138005 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "d889534999c34ff6af9802834ab5715f"
format_stamp: "Formatted at 2026-08-06 22:46:29 on dist-test-slave-zr1t"
server_key: "4c234dd3a801ff3d234dfd6b5756a2d8"
server_key_iv: "4eceb7ce90ee98091919559da3f6c5c1"
server_key_version: "encryptionkey@0"
I20260806 22:46:29.143044 22796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 22:46:29.146430 22814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:29.147694 22796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 22:46:29.147876 22796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "d889534999c34ff6af9802834ab5715f"
format_stamp: "Formatted at 2026-08-06 22:46:29 on dist-test-slave-zr1t"
server_key: "4c234dd3a801ff3d234dfd6b5756a2d8"
server_key_iv: "4eceb7ce90ee98091919559da3f6c5c1"
server_key_version: "encryptionkey@0"
I20260806 22:46:29.148074 22796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:29.167896 22796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:29.172328 22796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:29.172654 22796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:29.192787 22796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37173
I20260806 22:46:29.192823 22889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37173 every 8 connection(s)
I20260806 22:46:29.194242 22796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:29.195598 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 22796
I20260806 22:46:29.195796 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:29.196358 23954 external_mini_cluster.cc:1428] Setting key 660967f9822bd5170967d7417d7c88f2
I20260806 22:46:29.198971 22891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 22:46:29 dist-test-slave-zr1t krb5kdc[22778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 22:46:29.212241 22891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f: Bootstrap starting.
I20260806 22:46:29.215389 22891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:29.216434 22891 log.cc:824] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:29.219620 22891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f: No bootstrap required, opened a new log
I20260806 22:46:29.219638 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:29.198226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51845 (local address 127.0.0.1:37173)
0806 22:46:29.199106 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:29.199121 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:29.199171 (+ 50us) server_negotiation.cc:407] Connection header received
0806 22:46:29.200750 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:29.200769 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:29.201142 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:29.201463 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:29.203433 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:29.204737 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:29.206620 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:29.207254 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:29.207384 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:29.212272 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:29.212280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:29.212294 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:29.212333 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:29.215438 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:29.216169 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:29.216176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:29.216181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:29.216354 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:29.216815 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:29.216820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:29.216822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:29.217638 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:29.217935 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:29.218343 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 22:46:29.218645 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":415,"threads_started":1}
I20260806 22:46:29.223523 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:29.228161 22891 raft_consensus.cc:348] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889534999c34ff6af9802834ab5715f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } }
I20260806 22:46:29.228569 22891 raft_consensus.cc:374] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:29.228685 22891 raft_consensus.cc:729] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d889534999c34ff6af9802834ab5715f, State: Initialized, Role: FOLLOWER
I20260806 22:46:29.229302 22891 consensus_queue.cc:260] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889534999c34ff6af9802834ab5715f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } }
I20260806 22:46:29.229588 22891 raft_consensus.cc:388] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:29.229755 22891 raft_consensus.cc:482] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:29.229944 22891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:29.232280 22891 raft_consensus.cc:504] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889534999c34ff6af9802834ab5715f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } }
I20260806 22:46:29.232790 22891 leader_election.cc:302] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d889534999c34ff6af9802834ab5715f; no voters:
I20260806 22:46:29.233541 22891 leader_election.cc:288] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:29.233676 22904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:29.235767 22891 sys_catalog.cc:564] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:29.238035 22904 raft_consensus.cc:686] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [term 1 LEADER]: Becoming Leader. State: Replica: d889534999c34ff6af9802834ab5715f, State: Running, Role: LEADER
I20260806 22:46:29.239228 22904 consensus_queue.cc:237] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889534999c34ff6af9802834ab5715f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } }
I20260806 22:46:29.243459 22905 sys_catalog.cc:455] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d889534999c34ff6af9802834ab5715f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889534999c34ff6af9802834ab5715f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } } }
I20260806 22:46:29.244086 22905 sys_catalog.cc:458] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:29.245412 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d889534999c34ff6af9802834ab5715f. Latest consensus state: current_term: 1 leader_uuid: "d889534999c34ff6af9802834ab5715f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889534999c34ff6af9802834ab5715f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37173 } } }
I20260806 22:46:29.245929 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:29.246276 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:29.227992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55187 (local address 127.0.0.1:37173)
0806 22:46:29.228189 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:29.228196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:29.228677 (+ 481us) server_negotiation.cc:407] Connection header received
0806 22:46:29.228864 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:29.228869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:29.228941 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:29.229052 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:29.230731 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:29.231556 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:29.232848 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:29.233403 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:29.233582 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:29.237284 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:29.237294 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:29.237305 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:29.237361 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:29.242124 (+ 4763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:29.243043 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:29.243049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:29.243052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:29.243267 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:29.243748 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:29.243752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:29.243755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:29.244173 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:29.244782 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:29.244986 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 22:46:29.245284 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 22:46:29.251699 22926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 22:46:29.251892 22926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 22:46:29.252079 22925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:29.254374 22925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:29.263453 22925 catalog_manager.cc:1347] Generated new cluster ID: 3f102d1187ad463f9290cc17f727dcb7
I20260806 22:46:29.263583 22925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:29.276963 22925 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:29.278895 22925 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:29.291760 22925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d889534999c34ff6af9802834ab5715f: Generated new TSK 0
I20260806 22:46:29.292791 22925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:29.306649 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 22796
I20260806 22:46:29.307884 22883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:29.401624 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 22796
2026-08-06T22:46:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:29 dist-test-slave-zr1t krb5kdc[22935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:29 dist-test-slave-zr1t krb5kdc[22935](info): set up 2 sockets
Aug 06 22:46:29 dist-test-slave-zr1t krb5kdc[22935](info): commencing operation
krb5kdc: starting...
W20260806 22:46:30.771790 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.327s user 0.007s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:30 dist-test-slave-zr1t krb5kdc[22935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:31.128744 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:31.479768 22953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:31.480118 22953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:31.480300 22953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:31.487900 22953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:31.488034 22953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:31.488123 22953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:31.488210 22953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:31.488303 22953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:31.499044 22953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:31.500569 22953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:31.502162 22953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:31.516758 22964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:31.516880 22962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:31.516880 22961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:31.518121 22953 server_base.cc:1034] running on GCE node
I20260806 22:46:31.519088 22953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:31.520818 22953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:31.522056 22953 hybrid_clock.cc:648] HybridClock initialized: now 1786056391521982 us; error 77 us; skew 500 ppm
I20260806 22:46:31.522583 22953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:31 dist-test-slave-zr1t krb5kdc[22935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:31.526243 22953 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:31.528031 22953 webserver.cc:460] Webserver started at http://127.0.0.1:38889/ using document root <none> and password file <none>
I20260806 22:46:31.528739 22953 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:31.528851 22953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:31.529120 22953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:31.532567 22953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "2c7a8ad1d7df4da292fa36ffccfd03bd"
format_stamp: "Formatted at 2026-08-06 22:46:31 on dist-test-slave-zr1t"
server_key: "7064c715282d57cd980c4634cec15e35"
server_key_iv: "6995bf49e4a4da4cd61edc2b7b388fd3"
server_key_version: "encryptionkey@0"
I20260806 22:46:31.533433 22953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "2c7a8ad1d7df4da292fa36ffccfd03bd"
format_stamp: "Formatted at 2026-08-06 22:46:31 on dist-test-slave-zr1t"
server_key: "7064c715282d57cd980c4634cec15e35"
server_key_iv: "6995bf49e4a4da4cd61edc2b7b388fd3"
server_key_version: "encryptionkey@0"
I20260806 22:46:31.539252 22953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 22:46:31.543422 22972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:31.545127 22953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 22:46:31.545439 22953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "2c7a8ad1d7df4da292fa36ffccfd03bd"
format_stamp: "Formatted at 2026-08-06 22:46:31 on dist-test-slave-zr1t"
server_key: "7064c715282d57cd980c4634cec15e35"
server_key_iv: "6995bf49e4a4da4cd61edc2b7b388fd3"
server_key_version: "encryptionkey@0"
I20260806 22:46:31.545753 22953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:31.563915 22953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:31.568948 22953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:31.569319 22953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:31.589965 22953 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34497
I20260806 22:46:31.590029 23037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34497 every 8 connection(s)
I20260806 22:46:31.591856 22953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:31.593508 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 22953
I20260806 22:46:31.593777 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:31.594246 23954 external_mini_cluster.cc:1428] Setting key 5a4eed3f02077de7b2266c1ee4eb741f
I20260806 22:46:31.596309 23038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:31.610268 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:31.596201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42565 (local address 127.0.0.1:34497)
0806 22:46:31.597102 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:31.597118 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:31.597169 (+ 51us) server_negotiation.cc:407] Connection header received
0806 22:46:31.599490 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:31.599519 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:31.600076 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:31.600565 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:31.601082 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:31.603200 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:31.604613 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:31.605705 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:31.605895 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:31.607194 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:31.607208 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:31.607231 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:31.608044 (+ 813us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:31.608131 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:31.608166 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:31.608278 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:31.608531 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 22:46:31.608837 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":410,"threads_started":1}
I20260806 22:46:31.611435 23038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd: Bootstrap starting.
I20260806 22:46:31.614276 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:31.615020 23038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:31.616298 23038 log.cc:824] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:31.619971 23038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd: No bootstrap required, opened a new log
I20260806 22:46:31.626984 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:31.619636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36669 (local address 127.0.0.1:34497)
0806 22:46:31.619889 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:31.619898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:31.620350 (+ 452us) server_negotiation.cc:407] Connection header received
0806 22:46:31.620544 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:31.620548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:31.620613 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:31.620697 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:31.621391 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:31.622316 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:31.624278 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:31.624793 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:31.624917 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:31.626155 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:31.626161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:31.626164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:31.626182 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:31.626238 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:31.626254 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:31.626334 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:31.626513 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 22:46:31.626596 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 22:46:31.630354 23038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7a8ad1d7df4da292fa36ffccfd03bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260806 22:46:31.630815 23038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:31.630949 23038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c7a8ad1d7df4da292fa36ffccfd03bd, State: Initialized, Role: FOLLOWER
I20260806 22:46:31.631623 23038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7a8ad1d7df4da292fa36ffccfd03bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260806 22:46:31.631889 23038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:31.632058 23038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:31.632256 23038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:31.634778 23038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7a8ad1d7df4da292fa36ffccfd03bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260806 22:46:31.635365 23038 leader_election.cc:302] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c7a8ad1d7df4da292fa36ffccfd03bd; no voters:
I20260806 22:46:31.635958 23038 leader_election.cc:288] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:31.636096 23054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:31.636811 23054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [term 1 LEADER]: Becoming Leader. State: Replica: 2c7a8ad1d7df4da292fa36ffccfd03bd, State: Running, Role: LEADER
I20260806 22:46:31.637390 23054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7a8ad1d7df4da292fa36ffccfd03bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260806 22:46:31.638305 23038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:31.640739 23056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c7a8ad1d7df4da292fa36ffccfd03bd. Latest consensus state: current_term: 1 leader_uuid: "2c7a8ad1d7df4da292fa36ffccfd03bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7a8ad1d7df4da292fa36ffccfd03bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } } }
I20260806 22:46:31.640863 23055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c7a8ad1d7df4da292fa36ffccfd03bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7a8ad1d7df4da292fa36ffccfd03bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } } }
I20260806 22:46:31.641130 23056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:31.641216 23055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:31.643347 23064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:31.648934 23064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:31.659655 23064 catalog_manager.cc:1347] Generated new cluster ID: c1c73b89631747e1843693fff4df67f0
I20260806 22:46:31.659807 23064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:31.667661 23064 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:31.669694 23064 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:31.678169 23064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c7a8ad1d7df4da292fa36ffccfd03bd: Generated new TSK 0
I20260806 22:46:31.679093 23064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:31.693252 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 22953
W20260806 22:46:31.694777 23031 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42565
W20260806 22:46:31.695672 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 22953: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:46:31.695850 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 22953
2026-08-06T22:46:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:31 dist-test-slave-zr1t krb5kdc[23091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:31 dist-test-slave-zr1t krb5kdc[23091](info): set up 2 sockets
Aug 06 22:46:31 dist-test-slave-zr1t krb5kdc[23091](info): commencing operation
krb5kdc: starting...
W20260806 22:46:33.055873 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.312s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:33 dist-test-slave-zr1t krb5kdc[23091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:33.399000 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:33.745383 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:33.745697 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:33.745800 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:33.753156 23108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:33.753269 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:33.753316 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:33.753387 23108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:33.753432 23108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:33.763177 23108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:33.764389 23108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:33.765631 23108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:33.777472 23119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:33.777451 23118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:33.777412 23121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:33.778379 23108 server_base.cc:1034] running on GCE node
I20260806 22:46:33.779183 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:33.780607 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:33.781863 23108 hybrid_clock.cc:648] HybridClock initialized: now 1786056393781830 us; error 53 us; skew 500 ppm
I20260806 22:46:33.782217 23108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:33 dist-test-slave-zr1t krb5kdc[23091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:33.792359 23108 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:33.793962 23108 webserver.cc:460] Webserver started at http://127.0.0.1:41797/ using document root <none> and password file <none>
I20260806 22:46:33.794649 23108 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:33.794797 23108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:33.795123 23108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:33.798133 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "7f3f665e80f34bd0b1ec737f7cf71ac5"
format_stamp: "Formatted at 2026-08-06 22:46:33 on dist-test-slave-zr1t"
server_key: "7b031d26a598953b44ed5b2014cce8a1"
server_key_iv: "e61bbeee65ea55525419a1d89606ce6b"
server_key_version: "encryptionkey@0"
I20260806 22:46:33.799202 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "7f3f665e80f34bd0b1ec737f7cf71ac5"
format_stamp: "Formatted at 2026-08-06 22:46:33 on dist-test-slave-zr1t"
server_key: "7b031d26a598953b44ed5b2014cce8a1"
server_key_iv: "e61bbeee65ea55525419a1d89606ce6b"
server_key_version: "encryptionkey@0"
I20260806 22:46:33.804520 23108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 22:46:33.807997 23134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:33.809650 23108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:46:33.809842 23108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "7f3f665e80f34bd0b1ec737f7cf71ac5"
format_stamp: "Formatted at 2026-08-06 22:46:33 on dist-test-slave-zr1t"
server_key: "7b031d26a598953b44ed5b2014cce8a1"
server_key_iv: "e61bbeee65ea55525419a1d89606ce6b"
server_key_version: "encryptionkey@0"
I20260806 22:46:33.810056 23108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:33.815907 23108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:33.820442 23108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:33.820755 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:33.839977 23108 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46013
I20260806 22:46:33.840027 23201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46013 every 8 connection(s)
I20260806 22:46:33.841334 23108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:33.842402 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 23108
I20260806 22:46:33.842602 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:33.843077 23954 external_mini_cluster.cc:1428] Setting key 5129370c8fb2bf116ec7710a3ee6c28b
I20260806 22:46:33.845887 23202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:33.852119 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:33.845032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54973 (local address 127.0.0.1:46013)
0806 22:46:33.845887 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:33.845904 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:33.845956 (+ 52us) server_negotiation.cc:407] Connection header received
0806 22:46:33.848306 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:33.848331 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:33.848777 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:33.849199 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:33.850190 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:33.850198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:33.850214 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:33.850233 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:33.850336 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:33.850379 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:33.850608 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:33.850729 (+ 121us) server_negotiation.cc:299] Negotiation successful
0806 22:46:33.850911 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":354,"threads_started":1}
I20260806 22:46:33.855732 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:33.859782 23202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5: Bootstrap starting.
I20260806 22:46:33.863757 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:33.861027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52117 (local address 127.0.0.1:46013)
0806 22:46:33.861240 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:33.861249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:33.861771 (+ 522us) server_negotiation.cc:407] Connection header received
0806 22:46:33.861943 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:33.861948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:33.861996 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:33.862075 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:33.863111 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:33.863116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:33.863119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:33.863133 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:33.863171 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:33.863186 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:33.863272 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:33.863415 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 22:46:33.863467 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 22:46:33.864014 23202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:33.865125 23202 log.cc:824] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:33.868036 23202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5: No bootstrap required, opened a new log
I20260806 22:46:33.875903 23202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3f665e80f34bd0b1ec737f7cf71ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } }
I20260806 22:46:33.876376 23202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:33.876475 23202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3f665e80f34bd0b1ec737f7cf71ac5, State: Initialized, Role: FOLLOWER
I20260806 22:46:33.877116 23202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3f665e80f34bd0b1ec737f7cf71ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } }
I20260806 22:46:33.877367 23202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:33.877525 23202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:33.877727 23202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:33.880049 23202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3f665e80f34bd0b1ec737f7cf71ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } }
I20260806 22:46:33.880545 23202 leader_election.cc:302] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f3f665e80f34bd0b1ec737f7cf71ac5; no voters:
I20260806 22:46:33.881198 23202 leader_election.cc:288] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:33.881313 23214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:33.883211 23202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:33.885241 23214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3f665e80f34bd0b1ec737f7cf71ac5, State: Running, Role: LEADER
I20260806 22:46:33.885988 23214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3f665e80f34bd0b1ec737f7cf71ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } }
I20260806 22:46:33.889477 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f3f665e80f34bd0b1ec737f7cf71ac5. Latest consensus state: current_term: 1 leader_uuid: "7f3f665e80f34bd0b1ec737f7cf71ac5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3f665e80f34bd0b1ec737f7cf71ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } } }
I20260806 22:46:33.890070 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:33.889503 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f3f665e80f34bd0b1ec737f7cf71ac5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3f665e80f34bd0b1ec737f7cf71ac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46013 } } }
I20260806 22:46:33.890753 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:33.894570 23236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:33.899626 23236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:33.908152 23236 catalog_manager.cc:1347] Generated new cluster ID: f8eb6b3511ce4be89633230a6b91efd6
I20260806 22:46:33.908262 23236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:33.943284 23236 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:33.943897 23236 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:33.959787 23236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f3f665e80f34bd0b1ec737f7cf71ac5: Generated new TSK 0
I20260806 22:46:33.960656 23236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:33.974902 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 23108
W20260806 22:46:33.976181 23197 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54973
W20260806 22:46:33.976915 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 23108: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:46:33.977070 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 23108
2026-08-06T22:46:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:34 dist-test-slave-zr1t krb5kdc[23242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:34 dist-test-slave-zr1t krb5kdc[23242](info): set up 2 sockets
Aug 06 22:46:34 dist-test-slave-zr1t krb5kdc[23242](info): commencing operation
krb5kdc: starting...
W20260806 22:46:35.321703 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.299s user 0.007s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:35 dist-test-slave-zr1t krb5kdc[23242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:35.672212 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:36.038026 23263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:36.038383 23263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:36.038517 23263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:36.046077 23263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:36.046226 23263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:36.046332 23263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:36.046465 23263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:36.046545 23263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:36.056622 23263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:36.058007 23263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:36.059331 23263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:36.071259 23270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:36.071979 23271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:36.072613 23263 server_base.cc:1034] running on GCE node
W20260806 22:46:36.072649 23273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:36.073601 23263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:36.075057 23263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:36.076293 23263 hybrid_clock.cc:648] HybridClock initialized: now 1786056396076251 us; error 43 us; skew 500 ppm
I20260806 22:46:36.076654 23263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:36 dist-test-slave-zr1t krb5kdc[23242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:36.079862 23263 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:36.081384 23263 webserver.cc:460] Webserver started at http://127.0.0.1:42235/ using document root <none> and password file <none>
I20260806 22:46:36.082134 23263 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:36.082301 23263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:36.082612 23263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:36.085773 23263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "a387891a6faa4716842a449cfe1c16ff"
format_stamp: "Formatted at 2026-08-06 22:46:36 on dist-test-slave-zr1t"
server_key: "150d5b3cd959fb01784886c2a53e5115"
server_key_iv: "76cad2ad462e0ba4b4e4423f0726b2ad"
server_key_version: "encryptionkey@0"
I20260806 22:46:36.086634 23263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "a387891a6faa4716842a449cfe1c16ff"
format_stamp: "Formatted at 2026-08-06 22:46:36 on dist-test-slave-zr1t"
server_key: "150d5b3cd959fb01784886c2a53e5115"
server_key_iv: "76cad2ad462e0ba4b4e4423f0726b2ad"
server_key_version: "encryptionkey@0"
I20260806 22:46:36.092512 23263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:46:36.096150 23282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:36.097559 23263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:46:36.097765 23263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "a387891a6faa4716842a449cfe1c16ff"
format_stamp: "Formatted at 2026-08-06 22:46:36 on dist-test-slave-zr1t"
server_key: "150d5b3cd959fb01784886c2a53e5115"
server_key_iv: "76cad2ad462e0ba4b4e4423f0726b2ad"
server_key_version: "encryptionkey@0"
I20260806 22:46:36.097982 23263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:36.126309 23263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:36.130595 23263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:36.130892 23263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:36.150609 23263 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44379
I20260806 22:46:36.150655 23355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44379 every 8 connection(s)
I20260806 22:46:36.152117 23263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:36.156064 23356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:36.156699 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 23263
I20260806 22:46:36.156927 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:36.157502 23954 external_mini_cluster.cc:1428] Setting key 3f277116f373d12b5262ace88f147b3f
I20260806 22:46:36.171929 23356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff: Bootstrap starting.
Aug 06 22:46:36 dist-test-slave-zr1t krb5kdc[23242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 22:46:36.175890 23356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:36.177060 23356 log.cc:824] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:36.180188 23356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff: No bootstrap required, opened a new log
I20260806 22:46:36.183696 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:36.159622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:58951 (local address 127.0.0.1:44379)
0806 22:46:36.160482 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:36.160494 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:36.160540 (+ 46us) server_negotiation.cc:407] Connection header received
0806 22:46:36.162769 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:36.162798 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:36.163247 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:36.163595 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:36.165624 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:36.166998 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:36.169113 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:36.169981 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:36.170129 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:36.175537 (+ 5408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:36.175546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:36.175563 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:36.175614 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:36.178994 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:36.179968 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:36.179975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:36.179982 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:36.180163 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:36.180702 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:36.180705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:36.180708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:36.181275 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:36.181659 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:36.182284 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 22:46:36.182635 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":393,"threads_started":1}
I20260806 22:46:36.188349 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:36.189393 23356 raft_consensus.cc:348] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387891a6faa4716842a449cfe1c16ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260806 22:46:36.189858 23356 raft_consensus.cc:374] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:36.189968 23356 raft_consensus.cc:729] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a387891a6faa4716842a449cfe1c16ff, State: Initialized, Role: FOLLOWER
I20260806 22:46:36.190544 23356 consensus_queue.cc:260] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387891a6faa4716842a449cfe1c16ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260806 22:46:36.190800 23356 raft_consensus.cc:388] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:36.190953 23356 raft_consensus.cc:482] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:36.191133 23356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:36.193674 23356 raft_consensus.cc:504] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387891a6faa4716842a449cfe1c16ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260806 22:46:36.194339 23356 leader_election.cc:302] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a387891a6faa4716842a449cfe1c16ff; no voters:
I20260806 22:46:36.195106 23356 leader_election.cc:288] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:36.195220 23368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:36.196738 23368 raft_consensus.cc:686] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [term 1 LEADER]: Becoming Leader. State: Replica: a387891a6faa4716842a449cfe1c16ff, State: Running, Role: LEADER
I20260806 22:46:36.197464 23368 consensus_queue.cc:237] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387891a6faa4716842a449cfe1c16ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } }
I20260806 22:46:36.197765 23356 sys_catalog.cc:564] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:36.201789 23370 sys_catalog.cc:455] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a387891a6faa4716842a449cfe1c16ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387891a6faa4716842a449cfe1c16ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } } }
I20260806 22:46:36.202275 23370 sys_catalog.cc:458] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:36.202885 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader a387891a6faa4716842a449cfe1c16ff. Latest consensus state: current_term: 1 leader_uuid: "a387891a6faa4716842a449cfe1c16ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387891a6faa4716842a449cfe1c16ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44379 } } }
I20260806 22:46:36.203281 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:36.206501 23382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:36.211865 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:36.193192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:41207 (local address 127.0.0.1:44379)
0806 22:46:36.193405 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:36.193414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:36.194005 (+ 591us) server_negotiation.cc:407] Connection header received
0806 22:46:36.194183 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:36.194189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:36.194273 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:36.194394 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:36.196441 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:36.197484 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:36.198892 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:36.199823 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:36.199996 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:36.203714 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:36.203721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:36.203728 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:36.203768 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:36.207524 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:36.208359 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:36.208368 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:36.208373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:36.208504 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:36.209107 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:36.209114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:36.209119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:36.209642 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:36.211022 (+ 1380us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:36.211233 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 22:46:36.211443 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 22:46:36.213497 23382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:36.225171 23382 catalog_manager.cc:1347] Generated new cluster ID: f8a1ffc5b072449a8af52add0fe5d1c9
I20260806 22:46:36.225325 23382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:36.237905 23382 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:36.239202 23382 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:36.251806 23382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a387891a6faa4716842a449cfe1c16ff: Generated new TSK 0
I20260806 22:46:36.252791 23382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:36.264887 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 23263
I20260806 22:46:36.266839 23349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:36.359401 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 23263
W20260806 22:46:36.383318 23240 connection.cc:535] client connection to 127.0.0.1:44379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44379): Connection reset by peer (error 104)
2026-08-06T22:46:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:36 dist-test-slave-zr1t krb5kdc[23405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:36 dist-test-slave-zr1t krb5kdc[23405](info): set up 2 sockets
Aug 06 22:46:36 dist-test-slave-zr1t krb5kdc[23405](info): commencing operation
krb5kdc: starting...
W20260806 22:46:37.728864 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:37 dist-test-slave-zr1t krb5kdc[23405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:38.101155 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:38.448915 23420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:38.449268 23420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:38.449447 23420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:38.456640 23420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:38.456797 23420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:38.456909 23420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:38.457027 23420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:38.457106 23420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:38.467084 23420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:38.468303 23420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:38.469573 23420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:38.481304 23429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:38.481367 23431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:38.481258 23434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:38.482465 23420 server_base.cc:1034] running on GCE node
I20260806 22:46:38.483256 23420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:38.484661 23420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:38.485929 23420 hybrid_clock.cc:648] HybridClock initialized: now 1786056398485890 us; error 51 us; skew 500 ppm
I20260806 22:46:38.486409 23420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:38 dist-test-slave-zr1t krb5kdc[23405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:38.489799 23420 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:38.491323 23420 webserver.cc:460] Webserver started at http://127.0.0.1:33915/ using document root <none> and password file <none>
I20260806 22:46:38.492010 23420 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:38.492163 23420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:38.492478 23420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:38.495520 23420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "0618b3aad9f04240aff1a3adedffded6"
format_stamp: "Formatted at 2026-08-06 22:46:38 on dist-test-slave-zr1t"
server_key: "ef527ae4949d75582be9f05fbd356b36"
server_key_iv: "495701ac8796521f4b23889b8e68e059"
server_key_version: "encryptionkey@0"
I20260806 22:46:38.496315 23420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "0618b3aad9f04240aff1a3adedffded6"
format_stamp: "Formatted at 2026-08-06 22:46:38 on dist-test-slave-zr1t"
server_key: "ef527ae4949d75582be9f05fbd356b36"
server_key_iv: "495701ac8796521f4b23889b8e68e059"
server_key_version: "encryptionkey@0"
I20260806 22:46:38.501488 23420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 22:46:38.505287 23443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:38.506771 23420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:46:38.506958 23420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "0618b3aad9f04240aff1a3adedffded6"
format_stamp: "Formatted at 2026-08-06 22:46:38 on dist-test-slave-zr1t"
server_key: "ef527ae4949d75582be9f05fbd356b36"
server_key_iv: "495701ac8796521f4b23889b8e68e059"
server_key_version: "encryptionkey@0"
I20260806 22:46:38.507146 23420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:38.537091 23420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:38.541396 23420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:38.541678 23420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:38.560935 23420 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40321
I20260806 22:46:38.560979 23514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40321 every 8 connection(s)
I20260806 22:46:38.562376 23420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:38.565501 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 23420
I20260806 22:46:38.565721 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:38.566318 23954 external_mini_cluster.cc:1428] Setting key c57850cebeb75f7201c3da75971f411c
I20260806 22:46:38.566689 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:38.581789 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:38.568217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:32771 (local address 127.0.0.1:40321)
0806 22:46:38.569044 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:38.569060 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:38.569114 (+ 54us) server_negotiation.cc:407] Connection header received
0806 22:46:38.571561 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:38.571590 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:38.572150 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:38.572600 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:38.573131 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:38.575170 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:38.576577 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:38.577644 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:38.577850 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:38.579051 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:38.579066 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:38.579084 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:38.579822 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:38.579925 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:38.579956 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:38.580095 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:38.580341 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 22:46:38.580506 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":339,"threads_started":1}
I20260806 22:46:38.582072 23515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6: Bootstrap starting.
I20260806 22:46:38.585253 23515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:38.585634 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:38.586570 23515 log.cc:824] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:38.589908 23515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6: No bootstrap required, opened a new log
I20260806 22:46:38.597669 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:38.590792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:53671 (local address 127.0.0.1:40321)
0806 22:46:38.591028 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:38.591037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:38.591391 (+ 354us) server_negotiation.cc:407] Connection header received
0806 22:46:38.591588 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:38.591592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:38.591655 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:38.591740 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:38.592377 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:38.593161 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:38.595068 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:38.595524 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:38.595620 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:38.596823 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:38.596827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:38.596830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:38.596846 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:38.596898 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:38.596913 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:38.597002 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:38.597224 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 22:46:38.597300 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 22:46:38.600155 23515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0618b3aad9f04240aff1a3adedffded6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } }
I20260806 22:46:38.600533 23515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:38.600651 23515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0618b3aad9f04240aff1a3adedffded6, State: Initialized, Role: FOLLOWER
I20260806 22:46:38.601245 23515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0618b3aad9f04240aff1a3adedffded6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } }
I20260806 22:46:38.601501 23515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:38.601683 23515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:38.601854 23515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:38.604455 23515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0618b3aad9f04240aff1a3adedffded6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } }
I20260806 22:46:38.604974 23515 leader_election.cc:302] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0618b3aad9f04240aff1a3adedffded6; no voters:
I20260806 22:46:38.605501 23515 leader_election.cc:288] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:38.605716 23533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:38.606417 23533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [term 1 LEADER]: Becoming Leader. State: Replica: 0618b3aad9f04240aff1a3adedffded6, State: Running, Role: LEADER
I20260806 22:46:38.606909 23533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0618b3aad9f04240aff1a3adedffded6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } }
I20260806 22:46:38.607645 23515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:38.609828 23534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0618b3aad9f04240aff1a3adedffded6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0618b3aad9f04240aff1a3adedffded6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } } }
I20260806 22:46:38.609771 23535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0618b3aad9f04240aff1a3adedffded6. Latest consensus state: current_term: 1 leader_uuid: "0618b3aad9f04240aff1a3adedffded6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0618b3aad9f04240aff1a3adedffded6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40321 } } }
I20260806 22:46:38.610268 23535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:38.610909 23534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:38.612874 23544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:38.618093 23544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:38.627241 23544 catalog_manager.cc:1347] Generated new cluster ID: 254bd59d4b4a4ecb9d9cab7a987b2c9e
I20260806 22:46:38.627394 23544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:38.638813 23544 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:38.640077 23544 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:38.645874 23544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0618b3aad9f04240aff1a3adedffded6: Generated new TSK 0
I20260806 22:46:38.646904 23544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:38.658308 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 23420
W20260806 22:46:38.660220 23509 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.145:32771
W20260806 22:46:38.661067 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 23420: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:46:38.661228 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 23420
W20260806 22:46:38.689090 23403 connection.cc:535] client connection to 127.0.0.1:40321 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40321): Connection reset by peer (error 104)
2026-08-06T22:46:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:38 dist-test-slave-zr1t krb5kdc[23563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:38 dist-test-slave-zr1t krb5kdc[23563](info): set up 2 sockets
Aug 06 22:46:38 dist-test-slave-zr1t krb5kdc[23563](info): commencing operation
krb5kdc: starting...
W20260806 22:46:40.009384 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.300s user 0.003s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:40 dist-test-slave-zr1t krb5kdc[23563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:40.366652 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:40.720505 23584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:40.720806 23584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:40.720969 23584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:40.728149 23584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:40.728257 23584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:40.728379 23584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:40.728461 23584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:40.728546 23584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:40.738509 23584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:40.739867 23584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:40.741142 23584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:40.752936 23591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:40.752912 23595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:40.752863 23593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:40.754207 23584 server_base.cc:1034] running on GCE node
I20260806 22:46:40.755030 23584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:40.756476 23584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:40.757669 23584 hybrid_clock.cc:648] HybridClock initialized: now 1786056400757623 us; error 35 us; skew 500 ppm
I20260806 22:46:40.758031 23584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:40 dist-test-slave-zr1t krb5kdc[23563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:40.767979 23584 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:40.769734 23584 webserver.cc:460] Webserver started at http://127.0.0.1:35605/ using document root <none> and password file <none>
I20260806 22:46:40.770450 23584 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:40.770606 23584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:40.770926 23584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:40.774293 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "e893f59154cb482ba071914f64a5e08a"
format_stamp: "Formatted at 2026-08-06 22:46:40 on dist-test-slave-zr1t"
server_key: "6dc9625222e12c562b9dcc133077a67f"
server_key_iv: "295235898f4ae23bf7064a709b305b9f"
server_key_version: "encryptionkey@0"
I20260806 22:46:40.775118 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "e893f59154cb482ba071914f64a5e08a"
format_stamp: "Formatted at 2026-08-06 22:46:40 on dist-test-slave-zr1t"
server_key: "6dc9625222e12c562b9dcc133077a67f"
server_key_iv: "295235898f4ae23bf7064a709b305b9f"
server_key_version: "encryptionkey@0"
I20260806 22:46:40.780431 23584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:46:40.784246 23606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:40.785761 23584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:46:40.785977 23584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "e893f59154cb482ba071914f64a5e08a"
format_stamp: "Formatted at 2026-08-06 22:46:40 on dist-test-slave-zr1t"
server_key: "6dc9625222e12c562b9dcc133077a67f"
server_key_iv: "295235898f4ae23bf7064a709b305b9f"
server_key_version: "encryptionkey@0"
I20260806 22:46:40.786212 23584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:40.792750 23584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:40.797693 23584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:40.798038 23584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:40.819391 23584 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39343
I20260806 22:46:40.819446 23674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39343 every 8 connection(s)
I20260806 22:46:40.820936 23584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:40.821292 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 23584
I20260806 22:46:40.821475 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:40.822180 23954 external_mini_cluster.cc:1428] Setting key 47e3487808cb067c01b7e6391a5d8c55
I20260806 22:46:40.825440 23677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:40.831476 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:40.824439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:56921 (local address 127.0.0.1:39343)
0806 22:46:40.825422 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:40.825442 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:40.826178 (+ 736us) server_negotiation.cc:407] Connection header received
0806 22:46:40.827944 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:40.827966 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:40.828303 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:40.828655 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:40.829870 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:40.829876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:40.829889 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:40.829903 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:40.829984 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:40.830005 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:40.830188 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:40.830316 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 22:46:40.830524 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":431,"threads_started":1}
I20260806 22:46:40.835139 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:40.837703 23677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a: Bootstrap starting.
I20260806 22:46:40.841842 23677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:40.842841 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:40.840014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:42773 (local address 127.0.0.1:39343)
0806 22:46:40.840237 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:40.840289 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:40.840613 (+ 324us) server_negotiation.cc:407] Connection header received
0806 22:46:40.840787 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:40.840790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:40.840845 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:40.840929 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:40.842116 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:40.842120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:40.842124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:40.842139 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:40.842178 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:40.842193 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:40.842301 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:40.842483 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 22:46:40.842532 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 22:46:40.843320 23677 log.cc:824] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:40.846273 23677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a: No bootstrap required, opened a new log
I20260806 22:46:40.853713 23677 raft_consensus.cc:348] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e893f59154cb482ba071914f64a5e08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20260806 22:46:40.854127 23677 raft_consensus.cc:374] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:40.854257 23677 raft_consensus.cc:729] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e893f59154cb482ba071914f64a5e08a, State: Initialized, Role: FOLLOWER
I20260806 22:46:40.854938 23677 consensus_queue.cc:260] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e893f59154cb482ba071914f64a5e08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20260806 22:46:40.855283 23677 raft_consensus.cc:388] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:40.855444 23677 raft_consensus.cc:482] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:40.855640 23677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:40.857966 23677 raft_consensus.cc:504] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e893f59154cb482ba071914f64a5e08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20260806 22:46:40.858523 23677 leader_election.cc:302] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e893f59154cb482ba071914f64a5e08a; no voters:
I20260806 22:46:40.859119 23677 leader_election.cc:288] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:40.859290 23691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:40.860044 23691 raft_consensus.cc:686] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [term 1 LEADER]: Becoming Leader. State: Replica: e893f59154cb482ba071914f64a5e08a, State: Running, Role: LEADER
I20260806 22:46:40.860644 23691 consensus_queue.cc:237] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e893f59154cb482ba071914f64a5e08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } }
I20260806 22:46:40.861333 23677 sys_catalog.cc:564] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:40.863669 23696 sys_catalog.cc:455] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e893f59154cb482ba071914f64a5e08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e893f59154cb482ba071914f64a5e08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } } }
I20260806 22:46:40.864137 23696 sys_catalog.cc:458] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:40.865676 23697 sys_catalog.cc:455] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e893f59154cb482ba071914f64a5e08a. Latest consensus state: current_term: 1 leader_uuid: "e893f59154cb482ba071914f64a5e08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e893f59154cb482ba071914f64a5e08a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39343 } } }
I20260806 22:46:40.866083 23697 sys_catalog.cc:458] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:40.868305 23702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:40.873600 23702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:40.883011 23702 catalog_manager.cc:1347] Generated new cluster ID: b207ee1d9f2f41c5bed2fb434ee596d5
I20260806 22:46:40.883165 23702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:40.906754 23702 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:40.907222 23702 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:40.916054 23702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e893f59154cb482ba071914f64a5e08a: Generated new TSK 0
I20260806 22:46:40.917222 23702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:40.929505 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 23584
W20260806 22:46:40.930670 23669 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.145:56921
W20260806 22:46:40.931430 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 23584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:46:40.931607 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 23584
2026-08-06T22:46:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:41 dist-test-slave-zr1t krb5kdc[23721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:41 dist-test-slave-zr1t krb5kdc[23721](info): set up 2 sockets
Aug 06 22:46:41 dist-test-slave-zr1t krb5kdc[23721](info): commencing operation
krb5kdc: starting...
W20260806 22:46:42.277475 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.001s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:42 dist-test-slave-zr1t krb5kdc[23721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:42.642980 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:42.998873 23737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:42.999238 23737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:42.999404 23737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:43.007035 23737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:43.007212 23737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:43.007339 23737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:43.007462 23737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:43.007545 23737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:43.017953 23737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:43.019367 23737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:43.020749 23737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:43.032958 23744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:43.032985 23751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:43.033262 23748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:43.033955 23737 server_base.cc:1034] running on GCE node
I20260806 22:46:43.034888 23737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:43.036368 23737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:43.037644 23737 hybrid_clock.cc:648] HybridClock initialized: now 1786056403037600 us; error 59 us; skew 500 ppm
I20260806 22:46:43.038033 23737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:43 dist-test-slave-zr1t krb5kdc[23721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:43.041471 23737 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:43.043479 23737 webserver.cc:460] Webserver started at http://127.0.0.1:42385/ using document root <none> and password file <none>
I20260806 22:46:43.044602 23737 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:43.044785 23737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:43.045270 23737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:43.050056 23737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "4608e7241a664841a4bc45318bdb2b60"
format_stamp: "Formatted at 2026-08-06 22:46:43 on dist-test-slave-zr1t"
server_key: "1d8a132d3da239a59b02f872676498b5"
server_key_iv: "3e5367cdb28c33a05f2a5c76c9c79e76"
server_key_version: "encryptionkey@0"
I20260806 22:46:43.051138 23737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "4608e7241a664841a4bc45318bdb2b60"
format_stamp: "Formatted at 2026-08-06 22:46:43 on dist-test-slave-zr1t"
server_key: "1d8a132d3da239a59b02f872676498b5"
server_key_iv: "3e5367cdb28c33a05f2a5c76c9c79e76"
server_key_version: "encryptionkey@0"
I20260806 22:46:43.059031 23737 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260806 22:46:43.064775 23759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:43.068224 23737 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260806 22:46:43.068427 23737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "4608e7241a664841a4bc45318bdb2b60"
format_stamp: "Formatted at 2026-08-06 22:46:43 on dist-test-slave-zr1t"
server_key: "1d8a132d3da239a59b02f872676498b5"
server_key_iv: "3e5367cdb28c33a05f2a5c76c9c79e76"
server_key_version: "encryptionkey@0"
I20260806 22:46:43.068670 23737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:43.090926 23737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:43.095613 23737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:43.095862 23737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:43.115597 23737 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45479
I20260806 22:46:43.115628 23824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45479 every 8 connection(s)
I20260806 22:46:43.117079 23737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:43.118985 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 23737
I20260806 22:46:43.119184 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:43.119630 23954 external_mini_cluster.cc:1428] Setting key 37a039071788138fb128d2584d4eb29f
I20260806 22:46:43.121263 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 22:46:43 dist-test-slave-zr1t krb5kdc[23721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 22:46:43.135325 23825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60: Bootstrap starting.
I20260806 22:46:43.139732 23825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:43.141028 23825 log.cc:824] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:43.144093 23825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60: No bootstrap required, opened a new log
I20260806 22:46:43.144680 23828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:43.121377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:52809 (local address 127.0.0.1:45479)
0806 22:46:43.122269 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:43.122280 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:43.122322 (+ 42us) server_negotiation.cc:407] Connection header received
0806 22:46:43.123922 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:43.123942 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:43.124328 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:43.124652 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:43.126533 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:43.127934 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:43.129767 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:43.130466 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:43.130593 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:43.135754 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:43.135766 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:43.135785 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:43.135833 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:43.139995 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:43.140827 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:43.140836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:43.140846 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:43.141077 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:43.141759 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:43.141763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:43.141765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:43.142358 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:43.142712 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:43.143291 (+ 579us) server_negotiation.cc:299] Negotiation successful
0806 22:46:43.143609 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":447,"threads_started":1}
I20260806 22:46:43.148733 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:43.152266 23825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4608e7241a664841a4bc45318bdb2b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } }
I20260806 22:46:43.152683 23825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:43.152793 23825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4608e7241a664841a4bc45318bdb2b60, State: Initialized, Role: FOLLOWER
I20260806 22:46:43.153537 23825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4608e7241a664841a4bc45318bdb2b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } }
I20260806 22:46:43.153925 23825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:43.154081 23825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:43.154285 23825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:43.157431 23825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4608e7241a664841a4bc45318bdb2b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } }
I20260806 22:46:43.158129 23825 leader_election.cc:302] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4608e7241a664841a4bc45318bdb2b60; no voters:
I20260806 22:46:43.158813 23825 leader_election.cc:288] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:43.158939 23835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:43.160930 23825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:43.161549 23835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [term 1 LEADER]: Becoming Leader. State: Replica: 4608e7241a664841a4bc45318bdb2b60, State: Running, Role: LEADER
I20260806 22:46:43.162184 23835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4608e7241a664841a4bc45318bdb2b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } }
I20260806 22:46:43.166775 23839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4608e7241a664841a4bc45318bdb2b60. Latest consensus state: current_term: 1 leader_uuid: "4608e7241a664841a4bc45318bdb2b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4608e7241a664841a4bc45318bdb2b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } } }
I20260806 22:46:43.167189 23839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:43.167819 23837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4608e7241a664841a4bc45318bdb2b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4608e7241a664841a4bc45318bdb2b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45479 } } }
I20260806 22:46:43.168099 23837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:43.171998 23855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:43.174662 23828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:43.153546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:40595 (local address 127.0.0.1:45479)
0806 22:46:43.153784 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:43.153801 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:43.154233 (+ 432us) server_negotiation.cc:407] Connection header received
0806 22:46:43.154425 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:43.154431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:43.154510 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:43.154594 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:43.156428 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:43.157208 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:43.159149 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:43.160194 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:43.160367 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:43.166044 (+ 5677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:43.166051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:43.166056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:43.166094 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:43.170699 (+ 4605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:43.171955 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:43.171962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:43.171965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:43.172208 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:43.172817 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:43.172822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:43.172825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:43.173187 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:43.173499 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:43.174076 (+ 577us) server_negotiation.cc:299] Negotiation successful
0806 22:46:43.174261 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 22:46:43.185302 23855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:43.194605 23855 catalog_manager.cc:1347] Generated new cluster ID: 918f3c0eb6594ff9b6ead35cfa5cd365
I20260806 22:46:43.194754 23855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:43.222070 23855 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:43.223421 23855 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:43.237161 23855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4608e7241a664841a4bc45318bdb2b60: Generated new TSK 0
I20260806 22:46:43.238360 23855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:43.254712 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 23737
I20260806 22:46:43.256294 23819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:43.353109 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 23737
W20260806 22:46:43.377746 23719 connection.cc:535] client connection to 127.0.0.1:45479 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45479): Connection reset by peer (error 104)
2026-08-06T22:46:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:43 dist-test-slave-zr1t krb5kdc[23868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:43 dist-test-slave-zr1t krb5kdc[23868](info): set up 2 sockets
Aug 06 22:46:43 dist-test-slave-zr1t krb5kdc[23868](info): commencing operation
krb5kdc: starting...
W20260806 22:46:44.716089 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.319s user 0.006s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:44 dist-test-slave-zr1t krb5kdc[23868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:45.074999 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:45.423126 23887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:45.423413 23887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:45.423610 23887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:45.431077 23887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:45.431213 23887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:45.431300 23887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:45.431423 23887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:45.431510 23887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:45.441612 23887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:45.442894 23887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:45.444216 23887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:45.456319 23894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:45.456372 23899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:45.456398 23896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:45.457202 23887 server_base.cc:1034] running on GCE node
I20260806 22:46:45.458076 23887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:45.459507 23887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:45.460745 23887 hybrid_clock.cc:648] HybridClock initialized: now 1786056405460711 us; error 48 us; skew 500 ppm
I20260806 22:46:45.461123 23887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:45 dist-test-slave-zr1t krb5kdc[23868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:45.464322 23887 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:45.465776 23887 webserver.cc:460] Webserver started at http://127.0.0.1:34931/ using document root <none> and password file <none>
I20260806 22:46:45.466475 23887 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:45.466629 23887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:45.466953 23887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:45.469945 23887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "ca179f19f1cf4b64996dae6281bb5cc3"
format_stamp: "Formatted at 2026-08-06 22:46:45 on dist-test-slave-zr1t"
server_key: "0aac06469abd314d8fcdb8a1a07fe065"
server_key_iv: "32ad1324e793fd2e034fc12add6c0513"
server_key_version: "encryptionkey@0"
I20260806 22:46:45.470713 23887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "ca179f19f1cf4b64996dae6281bb5cc3"
format_stamp: "Formatted at 2026-08-06 22:46:45 on dist-test-slave-zr1t"
server_key: "0aac06469abd314d8fcdb8a1a07fe065"
server_key_iv: "32ad1324e793fd2e034fc12add6c0513"
server_key_version: "encryptionkey@0"
I20260806 22:46:45.475922 23887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:46:45.479461 23908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:45.480826 23887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:46:45.481047 23887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "ca179f19f1cf4b64996dae6281bb5cc3"
format_stamp: "Formatted at 2026-08-06 22:46:45 on dist-test-slave-zr1t"
server_key: "0aac06469abd314d8fcdb8a1a07fe065"
server_key_iv: "32ad1324e793fd2e034fc12add6c0513"
server_key_version: "encryptionkey@0"
I20260806 22:46:45.481258 23887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:45.492152 23887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:45.496574 23887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:45.496917 23887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:45.516888 23887 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42787
I20260806 22:46:45.516903 23980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42787 every 8 connection(s)
I20260806 22:46:45.518411 23887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:45.520382 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 23887
I20260806 22:46:45.520602 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:45.521200 23954 external_mini_cluster.cc:1428] Setting key 20862c6cb0971b67a5e7928b8a55ca4f
I20260806 22:46:45.522951 23982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:45.534510 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:45.523134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:50103 (local address 127.0.0.1:42787)
0806 22:46:45.523890 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:45.523902 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:45.523945 (+ 43us) server_negotiation.cc:407] Connection header received
0806 22:46:45.525627 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:45.525651 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:45.526045 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:45.526460 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:45.527010 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:45.528410 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:45.529836 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:45.530531 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:45.530742 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:45.532148 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:45.532156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:45.532169 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:45.532754 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:45.532837 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:45.532864 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:45.533047 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:45.533319 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 22:46:45.533491 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":322,"threads_started":1}
I20260806 22:46:45.534909 23982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3: Bootstrap starting.
I20260806 22:46:45.538995 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:45.538997 23982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:45.540055 23982 log.cc:824] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:45.543056 23982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3: No bootstrap required, opened a new log
I20260806 22:46:45.551906 23982 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca179f19f1cf4b64996dae6281bb5cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260806 22:46:45.552460 23982 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:45.552572 23982 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca179f19f1cf4b64996dae6281bb5cc3, State: Initialized, Role: FOLLOWER
I20260806 22:46:45.553269 23982 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca179f19f1cf4b64996dae6281bb5cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260806 22:46:45.553606 23982 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:45.553783 23982 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:45.554035 23982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:45.556128 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:45.548774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:39969 (local address 127.0.0.1:42787)
0806 22:46:45.548997 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:45.549005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:45.549376 (+ 371us) server_negotiation.cc:407] Connection header received
0806 22:46:45.549608 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:45.549613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:45.549679 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:45.549771 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:45.550547 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:45.551360 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:45.553208 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:45.553794 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:45.553927 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:45.555222 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:45.555227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:45.555231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:45.555250 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:45.555309 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:45.555325 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:45.555458 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:45.555693 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 22:46:45.555774 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 22:46:45.557255 23982 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca179f19f1cf4b64996dae6281bb5cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260806 22:46:45.557991 23982 leader_election.cc:302] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca179f19f1cf4b64996dae6281bb5cc3; no voters:
I20260806 22:46:45.558692 23982 leader_election.cc:288] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:45.558866 23994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:45.561187 23982 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:45.561815 23994 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [term 1 LEADER]: Becoming Leader. State: Replica: ca179f19f1cf4b64996dae6281bb5cc3, State: Running, Role: LEADER
I20260806 22:46:45.562448 23994 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca179f19f1cf4b64996dae6281bb5cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260806 22:46:45.566049 23996 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca179f19f1cf4b64996dae6281bb5cc3. Latest consensus state: current_term: 1 leader_uuid: "ca179f19f1cf4b64996dae6281bb5cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca179f19f1cf4b64996dae6281bb5cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } } }
I20260806 22:46:45.566502 23996 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:45.567114 23995 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca179f19f1cf4b64996dae6281bb5cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca179f19f1cf4b64996dae6281bb5cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } } }
I20260806 22:46:45.567449 23995 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:45.567654 24009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:45.577756 24009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:45.578608 24019 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 22:46:45.587896 24009 catalog_manager.cc:1347] Generated new cluster ID: 06a07823a5884ed4ac0d99ced95591d1
I20260806 22:46:45.588039 24009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:45.599119 24009 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:45.601220 24009 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:45.611943 24009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca179f19f1cf4b64996dae6281bb5cc3: Generated new TSK 0
I20260806 22:46:45.612938 24009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:45.627514 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 23887
W20260806 22:46:45.628814 23974 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.145:50103
W20260806 22:46:45.629733 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 23887: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:46:45.629914 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 23887
W20260806 22:46:45.655778 23866 connection.cc:535] client connection to 127.0.0.1:42787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42787): Connection reset by peer (error 104)
2026-08-06T22:46:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:45 dist-test-slave-zr1t krb5kdc[24025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:45 dist-test-slave-zr1t krb5kdc[24025](info): set up 2 sockets
Aug 06 22:46:45 dist-test-slave-zr1t krb5kdc[24025](info): commencing operation
krb5kdc: starting...
W20260806 22:46:46.978778 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.303s user 0.004s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:47 dist-test-slave-zr1t krb5kdc[24025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 22:46:47.340405 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:47.686424 24045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:47.686728 24045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:47.686914 24045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:47.694335 24045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:47.694453 24045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:47.694559 24045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:47.694676 24045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:47.694766 24045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:47.704821 24045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:47.706218 24045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:47.707561 24045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:47.719650 24052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:47.719677 24054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:47.719709 24051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:47.720371 24045 server_base.cc:1034] running on GCE node
I20260806 22:46:47.721113 24045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:47.722651 24045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:47.723889 24045 hybrid_clock.cc:648] HybridClock initialized: now 1786056407723818 us; error 70 us; skew 500 ppm
I20260806 22:46:47.724363 24045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:47 dist-test-slave-zr1t krb5kdc[24025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:47.732844 24045 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 22:46:47.734638 24045 webserver.cc:460] Webserver started at http://127.0.0.1:38195/ using document root <none> and password file <none>
I20260806 22:46:47.735374 24045 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:47.735536 24045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:47.735857 24045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:47.739009 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "3a9fba7340a2479ba07fdab87417bc3b"
format_stamp: "Formatted at 2026-08-06 22:46:47 on dist-test-slave-zr1t"
server_key: "9fcdebbbfc6e52dab3eab1a9038a332f"
server_key_iv: "5281b9ff903813f68d85b760f124684f"
server_key_version: "encryptionkey@0"
I20260806 22:46:47.739801 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "3a9fba7340a2479ba07fdab87417bc3b"
format_stamp: "Formatted at 2026-08-06 22:46:47 on dist-test-slave-zr1t"
server_key: "9fcdebbbfc6e52dab3eab1a9038a332f"
server_key_iv: "5281b9ff903813f68d85b760f124684f"
server_key_version: "encryptionkey@0"
I20260806 22:46:47.745052 24045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 22:46:47.748549 24064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:47.749927 24045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:46:47.750109 24045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "3a9fba7340a2479ba07fdab87417bc3b"
format_stamp: "Formatted at 2026-08-06 22:46:47 on dist-test-slave-zr1t"
server_key: "9fcdebbbfc6e52dab3eab1a9038a332f"
server_key_iv: "5281b9ff903813f68d85b760f124684f"
server_key_version: "encryptionkey@0"
I20260806 22:46:47.750321 24045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:47.756083 24045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:47.760221 24045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:47.760538 24045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:47.784160 24045 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39305
I20260806 22:46:47.784229 24125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39305 every 8 connection(s)
I20260806 22:46:47.786136 24045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:47.791208 24126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:47.794431 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24045
I20260806 22:46:47.794652 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:47.795249 23954 external_mini_cluster.cc:1428] Setting key b5e7c191d64478f099c09b8329a01905
I20260806 22:46:47.807240 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:47.798776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:40815 (local address 127.0.0.1:39305)
0806 22:46:47.800143 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:47.800161 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:47.800217 (+ 56us) server_negotiation.cc:407] Connection header received
0806 22:46:47.802775 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:47.802811 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:47.803379 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:47.803933 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:47.805029 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:47.805037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:47.805056 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:47.805078 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:47.805189 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:47.805217 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:47.805488 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:47.805645 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 22:46:47.805870 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1080,"thread_start_us":801,"threads_started":1}
I20260806 22:46:47.808511 24126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b: Bootstrap starting.
I20260806 22:46:47.813184 24126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:47.813859 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:47.814668 24126 log.cc:824] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:47.819895 24126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b: No bootstrap required, opened a new log
I20260806 22:46:47.833175 24126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9fba7340a2479ba07fdab87417bc3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260806 22:46:47.833762 24126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:47.833911 24126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9fba7340a2479ba07fdab87417bc3b, State: Initialized, Role: FOLLOWER
I20260806 22:46:47.834726 24126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9fba7340a2479ba07fdab87417bc3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260806 22:46:47.835114 24126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:47.835290 24126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:47.835490 24126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:47.836009 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:47.832559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:40979 (local address 127.0.0.1:39305)
0806 22:46:47.832867 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:47.832902 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:47.833273 (+ 371us) server_negotiation.cc:407] Connection header received
0806 22:46:47.833484 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:47.833489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:47.833591 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:47.833723 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:47.835094 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:47.835103 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:47.835110 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:47.835134 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:47.835190 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:47.835214 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:47.835327 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:47.835600 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 22:46:47.835677 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 22:46:47.839586 24126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9fba7340a2479ba07fdab87417bc3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260806 22:46:47.840320 24126 leader_election.cc:302] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a9fba7340a2479ba07fdab87417bc3b; no voters:
I20260806 22:46:47.841106 24126 leader_election.cc:288] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:47.841190 24142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:47.842072 24142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [term 1 LEADER]: Becoming Leader. State: Replica: 3a9fba7340a2479ba07fdab87417bc3b, State: Running, Role: LEADER
I20260806 22:46:47.842710 24142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9fba7340a2479ba07fdab87417bc3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } }
I20260806 22:46:47.843801 24126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:47.846354 24144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a9fba7340a2479ba07fdab87417bc3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9fba7340a2479ba07fdab87417bc3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } } }
I20260806 22:46:47.846812 24144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:47.847656 24154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:47.846376 24145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a9fba7340a2479ba07fdab87417bc3b. Latest consensus state: current_term: 1 leader_uuid: "3a9fba7340a2479ba07fdab87417bc3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9fba7340a2479ba07fdab87417bc3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39305 } } }
I20260806 22:46:47.849565 24145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:47.855827 24154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:47.864638 24154 catalog_manager.cc:1347] Generated new cluster ID: a46add908e80470f9725b541ac0c000f
I20260806 22:46:47.864760 24154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:47.880506 24154 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:47.880951 24154 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:47.890316 24154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a9fba7340a2479ba07fdab87417bc3b: Generated new TSK 0
I20260806 22:46:47.891177 24154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:47.906713 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24045
W20260806 22:46:47.907927 24121 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.145:40815
W20260806 22:46:47.908689 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24045: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:46:47.908844 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24045
2026-08-06T22:46:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:48 dist-test-slave-zr1t krb5kdc[24173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:48 dist-test-slave-zr1t krb5kdc[24173](info): set up 2 sockets
Aug 06 22:46:48 dist-test-slave-zr1t krb5kdc[24173](info): commencing operation
krb5kdc: starting...
W20260806 22:46:49.257802 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.302s user 0.004s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:49 dist-test-slave-zr1t krb5kdc[24173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:46:49.608711 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:34853
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:34853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:49.962770 24192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:49.963073 24192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:49.963269 24192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:49.970625 24192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:49.970770 24192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:49.970863 24192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:49.970971 24192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:49.971055 24192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:49.981016 24192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:34853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:34853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:49.982456 24192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:49.983781 24192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:49.996133 24199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:49.996099 24202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:49.996053 24200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:49.997131 24192 server_base.cc:1034] running on GCE node
I20260806 22:46:49.998096 24192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:49.999606 24192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:50.000826 24192 hybrid_clock.cc:648] HybridClock initialized: now 1786056410000773 us; error 42 us; skew 500 ppm
I20260806 22:46:50.001216 24192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:50 dist-test-slave-zr1t krb5kdc[24173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056410, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:50.004576 24192 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:46:50.006194 24192 webserver.cc:460] Webserver started at http://127.23.100.190:40779/ using document root <none> and password file <none>
I20260806 22:46:50.006898 24192 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:50.007011 24192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:50.007362 24192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:50.011220 24192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "c230e0fc85fe4c41986a43846294a5f3"
format_stamp: "Formatted at 2026-08-06 22:46:50 on dist-test-slave-zr1t"
server_key: "af2923c64450ccd1201bfcbe2235bcdf"
server_key_iv: "bbd0e6a882a96b512fdf1751f47063dc"
server_key_version: "encryptionkey@0"
I20260806 22:46:50.012192 24192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "c230e0fc85fe4c41986a43846294a5f3"
format_stamp: "Formatted at 2026-08-06 22:46:50 on dist-test-slave-zr1t"
server_key: "af2923c64450ccd1201bfcbe2235bcdf"
server_key_iv: "bbd0e6a882a96b512fdf1751f47063dc"
server_key_version: "encryptionkey@0"
I20260806 22:46:50.017411 24192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 22:46:50.021049 24213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:50.022557 24192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:46:50.022760 24192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "c230e0fc85fe4c41986a43846294a5f3"
format_stamp: "Formatted at 2026-08-06 22:46:50 on dist-test-slave-zr1t"
server_key: "af2923c64450ccd1201bfcbe2235bcdf"
server_key_iv: "bbd0e6a882a96b512fdf1751f47063dc"
server_key_version: "encryptionkey@0"
I20260806 22:46:50.022984 24192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:50.038487 24192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:50.042752 24192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:50.043071 24192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:50.062932 24192 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:34853
I20260806 22:46:50.062984 24277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:34853 every 8 connection(s)
I20260806 22:46:50.064431 24192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:50.068220 24278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:50.073735 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24192
I20260806 22:46:50.073954 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:50.074440 23954 external_mini_cluster.cc:1428] Setting key 850309ec6e7ae6fb0a31d694081f96f5
I20260806 22:46:50.078999 24278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3: Bootstrap starting.
I20260806 22:46:50.082681 24278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:50.083835 24278 log.cc:824] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:50.087227 24278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3: No bootstrap required, opened a new log
Aug 06 22:46:50 dist-test-slave-zr1t krb5kdc[24173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:46:50.095573 24278 raft_consensus.cc:348] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230e0fc85fe4c41986a43846294a5f3" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34853 } }
I20260806 22:46:50.095981 24278 raft_consensus.cc:374] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:50.096092 24278 raft_consensus.cc:729] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c230e0fc85fe4c41986a43846294a5f3, State: Initialized, Role: FOLLOWER
I20260806 22:46:50.096678 24278 consensus_queue.cc:260] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230e0fc85fe4c41986a43846294a5f3" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34853 } }
I20260806 22:46:50.096946 24278 raft_consensus.cc:388] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:50.097105 24278 raft_consensus.cc:482] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:50.097316 24278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:50.099295 24283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:50.076425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46229 (local address 127.23.100.190:34853)
0806 22:46:50.077321 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:50.077333 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:50.077373 (+ 40us) server_negotiation.cc:407] Connection header received
0806 22:46:50.079111 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:50.079166 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:50.079581 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:50.079891 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:50.081857 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:50.083267 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:50.084936 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:50.085818 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:50.085951 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:50.091172 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:50.091180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:50.091196 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:50.091242 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:50.094658 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:50.095504 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:50.095512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:50.095519 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:50.095722 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:50.096279 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:50.096291 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:50.096294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:50.096972 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:50.097309 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:50.097840 (+ 531us) server_negotiation.cc:299] Negotiation successful
0806 22:46:50.098168 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":432,"threads_started":1}
I20260806 22:46:50.100059 24278 raft_consensus.cc:504] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230e0fc85fe4c41986a43846294a5f3" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34853 } }
I20260806 22:46:50.100593 24278 leader_election.cc:302] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c230e0fc85fe4c41986a43846294a5f3; no voters:
I20260806 22:46:50.101284 24278 leader_election.cc:288] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:50.101399 24285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:50.102809 24285 raft_consensus.cc:686] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [term 1 LEADER]: Becoming Leader. State: Replica: c230e0fc85fe4c41986a43846294a5f3, State: Running, Role: LEADER
I20260806 22:46:50.103391 24285 consensus_queue.cc:237] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230e0fc85fe4c41986a43846294a5f3" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34853 } }
I20260806 22:46:50.103708 24278 sys_catalog.cc:564] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:50.103803 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:50.106413 24289 sys_catalog.cc:455] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c230e0fc85fe4c41986a43846294a5f3. Latest consensus state: current_term: 1 leader_uuid: "c230e0fc85fe4c41986a43846294a5f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230e0fc85fe4c41986a43846294a5f3" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34853 } } }
I20260806 22:46:50.106933 24289 sys_catalog.cc:458] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:50.107559 24286 sys_catalog.cc:455] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c230e0fc85fe4c41986a43846294a5f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c230e0fc85fe4c41986a43846294a5f3" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 34853 } } }
I20260806 22:46:50.107890 24286 sys_catalog.cc:458] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:50.110952 24297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:50.117624 24297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:50.146185 24297 catalog_manager.cc:1347] Generated new cluster ID: a8d1472ae50f41fb89e1f0a0f623df33
I20260806 22:46:50.146358 24297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:50.162829 24283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:50.133454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53111 (local address 127.23.100.190:34853)
0806 22:46:50.133923 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:50.133933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:50.143800 (+ 9867us) server_negotiation.cc:407] Connection header received
0806 22:46:50.144337 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:50.144342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:50.144432 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:50.144513 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:50.146864 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:50.148122 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:50.150056 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:50.150819 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:50.150979 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:50.154754 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:50.154761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:50.154767 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:50.154815 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:50.158823 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:50.159933 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:50.159939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:50.159943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:50.160124 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:50.160748 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:50.160753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:50.160757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:50.161187 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:50.161445 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:50.162186 (+ 741us) server_negotiation.cc:299] Negotiation successful
0806 22:46:50.162414 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260806 22:46:50.166185 24297 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:50.168663 24297 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:50.189370 24297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c230e0fc85fe4c41986a43846294a5f3: Generated new TSK 0
I20260806 22:46:50.190554 24297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:50.200577 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24192
I20260806 22:46:50.202001 24273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:50.297794 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24192
W20260806 22:46:50.323845 24170 connection.cc:535] client connection to 127.23.100.190:34853 recv error: Network error: failed to read from TLS socket (remote: 127.23.100.190:34853): Connection reset by peer (error 104)
2026-08-06T22:46:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:50 dist-test-slave-zr1t krb5kdc[24328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:50 dist-test-slave-zr1t krb5kdc[24328](info): set up 2 sockets
Aug 06 22:46:50 dist-test-slave-zr1t krb5kdc[24328](info): commencing operation
krb5kdc: starting...
W20260806 22:46:51.638789 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:51 dist-test-slave-zr1t krb5kdc[24328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:46:51.991029 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:43167
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:33713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:43167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:52.349228 24348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:52.349602 24348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:52.349786 24348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:52.357805 24348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:52.357959 24348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:52.358064 24348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:52.358182 24348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:52.358268 24348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:52.368309 24348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:33713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:43167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:52.369688 24348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:52.370993 24348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:52.383044 24354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:52.382957 24358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:52.382999 24356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:52.384169 24348 server_base.cc:1034] running on GCE node
I20260806 22:46:52.385002 24348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:52.386566 24348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:52.387781 24348 hybrid_clock.cc:648] HybridClock initialized: now 1786056412387722 us; error 57 us; skew 500 ppm
I20260806 22:46:52.388159 24348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:52 dist-test-slave-zr1t krb5kdc[24328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056412, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:52.391660 24348 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:46:52.393224 24348 webserver.cc:460] Webserver started at http://127.23.100.190:36249/ using document root <none> and password file <none>
I20260806 22:46:52.393950 24348 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:52.394054 24348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:52.394305 24348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:52.397294 24348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "49a162350049401eb218763b00acc710"
format_stamp: "Formatted at 2026-08-06 22:46:52 on dist-test-slave-zr1t"
server_key: "14ae6547e003cfc4967e9c0ae1d2c571"
server_key_iv: "68ad2fe5d741136f94aed1d218b4d7da"
server_key_version: "encryptionkey@0"
I20260806 22:46:52.398111 24348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "49a162350049401eb218763b00acc710"
format_stamp: "Formatted at 2026-08-06 22:46:52 on dist-test-slave-zr1t"
server_key: "14ae6547e003cfc4967e9c0ae1d2c571"
server_key_iv: "68ad2fe5d741136f94aed1d218b4d7da"
server_key_version: "encryptionkey@0"
I20260806 22:46:52.403326 24348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 22:46:52.406838 24368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:52.408211 24348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:46:52.408443 24348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "49a162350049401eb218763b00acc710"
format_stamp: "Formatted at 2026-08-06 22:46:52 on dist-test-slave-zr1t"
server_key: "14ae6547e003cfc4967e9c0ae1d2c571"
server_key_iv: "68ad2fe5d741136f94aed1d218b4d7da"
server_key_version: "encryptionkey@0"
I20260806 22:46:52.408658 24348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:52.428756 24348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:52.433394 24348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:52.433738 24348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:52.452940 24348 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:43167
I20260806 22:46:52.452961 24431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:43167 every 8 connection(s)
I20260806 22:46:52.454315 24348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:52.455894 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24348
I20260806 22:46:52.456123 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:52.456641 23954 external_mini_cluster.cc:1428] Setting key 3e844f6dca29e5eebc54b620cbf8ef5b
I20260806 22:46:52.458566 24433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:52.471896 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:52.458591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35991 (local address 127.23.100.190:43167)
0806 22:46:52.459402 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:52.459417 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:52.459468 (+ 51us) server_negotiation.cc:407] Connection header received
0806 22:46:52.461689 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:52.461716 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:52.462226 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:52.462684 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:52.463199 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:52.465207 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:52.466585 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:52.467634 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:52.467820 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:52.469068 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:52.469077 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:52.469099 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:52.469866 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:52.469969 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:52.469999 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:52.470176 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:52.470439 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 22:46:52.470632 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":332,"threads_started":1}
I20260806 22:46:52.473241 24433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710: Bootstrap starting.
I20260806 22:46:52.476174 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:52.477017 24433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:52.478295 24433 log.cc:824] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:52.481429 24433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710: No bootstrap required, opened a new log
I20260806 22:46:52.488605 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:52.481622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47529 (local address 127.23.100.190:43167)
0806 22:46:52.481797 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:52.481803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:52.482193 (+ 390us) server_negotiation.cc:407] Connection header received
0806 22:46:52.482362 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:52.482367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:52.482450 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:52.482559 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:52.483085 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:52.484303 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:52.485729 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:52.486425 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:52.486567 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:52.487713 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:52.487720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:52.487725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:52.487748 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:52.487799 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:52.487818 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:52.487944 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:52.488198 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 22:46:52.488253 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 22:46:52.491371 24433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a162350049401eb218763b00acc710" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43167 } }
I20260806 22:46:52.491807 24433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:52.491941 24433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a162350049401eb218763b00acc710, State: Initialized, Role: FOLLOWER
I20260806 22:46:52.492506 24433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a162350049401eb218763b00acc710" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43167 } }
I20260806 22:46:52.492739 24433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:52.492880 24433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:52.493098 24433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:52.495560 24433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a162350049401eb218763b00acc710" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43167 } }
I20260806 22:46:52.496094 24433 leader_election.cc:302] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49a162350049401eb218763b00acc710; no voters:
I20260806 22:46:52.496685 24433 leader_election.cc:288] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:52.496841 24448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:52.497604 24448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [term 1 LEADER]: Becoming Leader. State: Replica: 49a162350049401eb218763b00acc710, State: Running, Role: LEADER
I20260806 22:46:52.498126 24448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a162350049401eb218763b00acc710" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43167 } }
I20260806 22:46:52.499084 24433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:52.501538 24450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49a162350049401eb218763b00acc710. Latest consensus state: current_term: 1 leader_uuid: "49a162350049401eb218763b00acc710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a162350049401eb218763b00acc710" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43167 } } }
I20260806 22:46:52.501937 24450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:52.501557 24449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49a162350049401eb218763b00acc710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49a162350049401eb218763b00acc710" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43167 } } }
I20260806 22:46:52.502306 24449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:52.505015 24462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:52.510402 24462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:52.519742 24462 catalog_manager.cc:1347] Generated new cluster ID: ef5d4f2d259043c0a7147064c6d0ef0b
I20260806 22:46:52.519858 24462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:52.541929 24462 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:52.543782 24462 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:52.558116 24462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49a162350049401eb218763b00acc710: Generated new TSK 0
I20260806 22:46:52.559183 24462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:52.565927 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24348
W20260806 22:46:52.567240 24427 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35991
W20260806 22:46:52.568073 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24348: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:46:52.568221 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24348
W20260806 22:46:52.596665 24326 connection.cc:535] client connection to 127.23.100.190:43167 recv error: Network error: failed to read from TLS socket (remote: 127.23.100.190:43167): Connection reset by peer (error 104)
2026-08-06T22:46:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:52 dist-test-slave-zr1t krb5kdc[24480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:52 dist-test-slave-zr1t krb5kdc[24480](info): set up 2 sockets
Aug 06 22:46:52 dist-test-slave-zr1t krb5kdc[24480](info): commencing operation
krb5kdc: starting...
W20260806 22:46:53.908675 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.001s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:54 dist-test-slave-zr1t krb5kdc[24480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:46:54.269498 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:45911
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:44441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:54.611581 24495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:54.611888 24495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:54.611984 24495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:54.619303 24495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:54.619428 24495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:54.619488 24495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:54.619552 24495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:54.619596 24495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:54.629426 24495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:44441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:45911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:54.630716 24495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:54.632010 24495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:54.643783 24506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:54.643836 24507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:54.643733 24510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:54.644843 24495 server_base.cc:1034] running on GCE node
I20260806 22:46:54.645684 24495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:54.647017 24495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:54.648219 24495 hybrid_clock.cc:648] HybridClock initialized: now 1786056414648178 us; error 30 us; skew 500 ppm
I20260806 22:46:54.648566 24495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:54 dist-test-slave-zr1t krb5kdc[24480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056414, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:54.656728 24495 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:46:54.658463 24495 webserver.cc:460] Webserver started at http://127.23.100.190:37871/ using document root <none> and password file <none>
I20260806 22:46:54.659179 24495 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:54.659317 24495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:54.659649 24495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:54.662775 24495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "f4f4e13fbe9b475190358337ee2d67cb"
format_stamp: "Formatted at 2026-08-06 22:46:54 on dist-test-slave-zr1t"
server_key: "7a777d743e53056e30bc63f90a0621df"
server_key_iv: "d04b11d2ea4da5838b98c80cf2661222"
server_key_version: "encryptionkey@0"
I20260806 22:46:54.663578 24495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "f4f4e13fbe9b475190358337ee2d67cb"
format_stamp: "Formatted at 2026-08-06 22:46:54 on dist-test-slave-zr1t"
server_key: "7a777d743e53056e30bc63f90a0621df"
server_key_iv: "d04b11d2ea4da5838b98c80cf2661222"
server_key_version: "encryptionkey@0"
I20260806 22:46:54.668895 24495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 22:46:54.672574 24518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:54.674017 24495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:46:54.674209 24495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "f4f4e13fbe9b475190358337ee2d67cb"
format_stamp: "Formatted at 2026-08-06 22:46:54 on dist-test-slave-zr1t"
server_key: "7a777d743e53056e30bc63f90a0621df"
server_key_iv: "d04b11d2ea4da5838b98c80cf2661222"
server_key_version: "encryptionkey@0"
I20260806 22:46:54.674417 24495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:54.680171 24495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:54.684520 24495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:54.684820 24495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:54.702667 24495 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:45911
I20260806 22:46:54.702690 24590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:45911 every 8 connection(s)
I20260806 22:46:54.704151 24495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:54.708015 24592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:54.713306 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24495
I20260806 22:46:54.713493 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:54.713970 23954 external_mini_cluster.cc:1428] Setting key 505d575e14792f441a9649d3202c0bf5
I20260806 22:46:54.719074 24592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb: Bootstrap starting.
I20260806 22:46:54.722016 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:54.715815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48941 (local address 127.23.100.190:45911)
0806 22:46:54.716760 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:54.716772 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:54.716820 (+ 48us) server_negotiation.cc:407] Connection header received
0806 22:46:54.718565 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:54.718587 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:54.718929 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:54.719305 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:54.720386 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:54.720393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:54.720407 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:54.720421 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:54.720488 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:54.720512 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:54.720753 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:54.720901 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 22:46:54.721092 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":442,"threads_started":1}
I20260806 22:46:54.722630 24592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:54.723857 24592 log.cc:824] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:54.726277 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:54.727362 24592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb: No bootstrap required, opened a new log
I20260806 22:46:54.733484 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:54.730805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36001 (local address 127.23.100.190:45911)
0806 22:46:54.731037 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:54.731046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:54.731423 (+ 377us) server_negotiation.cc:407] Connection header received
0806 22:46:54.731649 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:54.731661 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:54.731711 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:54.731828 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:54.732801 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:54.732814 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:54.732817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:54.732832 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:54.732873 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:54.732892 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:54.733043 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:54.733134 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 22:46:54.733205 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 22:46:54.736418 24592 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4e13fbe9b475190358337ee2d67cb" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45911 } }
I20260806 22:46:54.736971 24592 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:54.737105 24592 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f4e13fbe9b475190358337ee2d67cb, State: Initialized, Role: FOLLOWER
I20260806 22:46:54.737736 24592 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4e13fbe9b475190358337ee2d67cb" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45911 } }
I20260806 22:46:54.737984 24592 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:54.738150 24592 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:54.738329 24592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:54.740545 24592 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4e13fbe9b475190358337ee2d67cb" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45911 } }
I20260806 22:46:54.741065 24592 leader_election.cc:302] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4f4e13fbe9b475190358337ee2d67cb; no voters:
I20260806 22:46:54.741727 24592 leader_election.cc:288] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:54.741856 24608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:54.742547 24608 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [term 1 LEADER]: Becoming Leader. State: Replica: f4f4e13fbe9b475190358337ee2d67cb, State: Running, Role: LEADER
I20260806 22:46:54.743090 24608 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4e13fbe9b475190358337ee2d67cb" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45911 } }
I20260806 22:46:54.743834 24592 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:54.745987 24611 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4f4e13fbe9b475190358337ee2d67cb. Latest consensus state: current_term: 1 leader_uuid: "f4f4e13fbe9b475190358337ee2d67cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4e13fbe9b475190358337ee2d67cb" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45911 } } }
I20260806 22:46:54.746472 24611 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:54.748355 24610 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4f4e13fbe9b475190358337ee2d67cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f4e13fbe9b475190358337ee2d67cb" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 45911 } } }
I20260806 22:46:54.748776 24610 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:54.749379 24617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:54.754593 24617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:54.763697 24617 catalog_manager.cc:1347] Generated new cluster ID: 0cd9e3e7bf1f4c349f7e3810e8a57e1e
I20260806 22:46:54.763811 24617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:54.785353 24617 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:54.785831 24617 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:54.802861 24617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4f4e13fbe9b475190358337ee2d67cb: Generated new TSK 0
I20260806 22:46:54.803752 24617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:54.811553 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24495
W20260806 22:46:54.812772 24586 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48941
W20260806 22:46:54.813556 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24495: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:46:54.813721 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24495
2026-08-06T22:46:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:54 dist-test-slave-zr1t krb5kdc[24638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:54 dist-test-slave-zr1t krb5kdc[24638](info): set up 2 sockets
Aug 06 22:46:54 dist-test-slave-zr1t krb5kdc[24638](info): commencing operation
krb5kdc: starting...
W20260806 22:46:56.176703 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.316s user 0.004s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:56 dist-test-slave-zr1t krb5kdc[24638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:46:56.526909 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33863
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:42515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:33863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:56.880614 24658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:56.880911 24658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:56.881104 24658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:56.888497 24658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:56.888612 24658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:56.888659 24658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:56.888758 24658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:56.888829 24658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:56.898895 24658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:42515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:33863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:56.900638 24658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:56.902125 24658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:56.914506 24665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:56.914484 24666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:56.914564 24668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:56.915299 24658 server_base.cc:1034] running on GCE node
I20260806 22:46:56.916100 24658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:56.917557 24658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:56.918813 24658 hybrid_clock.cc:648] HybridClock initialized: now 1786056416918766 us; error 52 us; skew 500 ppm
I20260806 22:46:56.919194 24658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:56 dist-test-slave-zr1t krb5kdc[24638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056416, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:56.922503 24658 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:46:56.924081 24658 webserver.cc:460] Webserver started at http://127.23.100.190:37499/ using document root <none> and password file <none>
I20260806 22:46:56.924780 24658 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:56.924945 24658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:56.925256 24658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:56.928470 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "79b5cc1795dd46888c6a20c5ece129f5"
format_stamp: "Formatted at 2026-08-06 22:46:56 on dist-test-slave-zr1t"
server_key: "12bf3bfbe8e9545018a1512858848f64"
server_key_iv: "80bd8d24fc948b34d562267f0ac199fd"
server_key_version: "encryptionkey@0"
I20260806 22:46:56.929282 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "79b5cc1795dd46888c6a20c5ece129f5"
format_stamp: "Formatted at 2026-08-06 22:46:56 on dist-test-slave-zr1t"
server_key: "12bf3bfbe8e9545018a1512858848f64"
server_key_iv: "80bd8d24fc948b34d562267f0ac199fd"
server_key_version: "encryptionkey@0"
I20260806 22:46:56.934571 24658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 22:46:56.938277 24677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:56.939679 24658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 22:46:56.939889 24658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "79b5cc1795dd46888c6a20c5ece129f5"
format_stamp: "Formatted at 2026-08-06 22:46:56 on dist-test-slave-zr1t"
server_key: "12bf3bfbe8e9545018a1512858848f64"
server_key_iv: "80bd8d24fc948b34d562267f0ac199fd"
server_key_version: "encryptionkey@0"
I20260806 22:46:56.940115 24658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:56.961082 24658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:56.965488 24658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:56.965828 24658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:56.986183 24658 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:33863
I20260806 22:46:56.986204 24744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:33863 every 8 connection(s)
I20260806 22:46:56.987725 24658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:56.991457 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:56.992096 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24658
I20260806 22:46:56.992300 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:56.992936 23954 external_mini_cluster.cc:1428] Setting key 389511d1c2c37e7a328b7b0272aea54e
I20260806 22:46:57.005940 24745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5: Bootstrap starting.
Aug 06 22:46:57 dist-test-slave-zr1t krb5kdc[24638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:46:57.009719 24745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:57.010777 24745 log.cc:824] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:57.013665 24745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5: No bootstrap required, opened a new log
I20260806 22:46:57.017282 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:56.994932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42197 (local address 127.23.100.190:33863)
0806 22:46:56.995784 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:56.995796 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:56.995842 (+ 46us) server_negotiation.cc:407] Connection header received
0806 22:46:56.997951 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:56.997974 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:56.998384 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:56.998733 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:57.000604 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:57.002148 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:57.003989 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:57.004647 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:57.004780 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:57.009723 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:57.009731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:57.009745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:57.009787 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:57.013060 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:57.013835 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:57.013848 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:57.013854 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:57.014030 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:57.014469 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:57.014475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:57.014478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:57.014997 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:57.015379 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:57.015828 (+ 449us) server_negotiation.cc:299] Negotiation successful
0806 22:46:57.016287 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":395,"threads_started":1}
I20260806 22:46:57.020989 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:57.021937 24745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b5cc1795dd46888c6a20c5ece129f5" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33863 } }
I20260806 22:46:57.022325 24745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:57.022433 24745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b5cc1795dd46888c6a20c5ece129f5, State: Initialized, Role: FOLLOWER
I20260806 22:46:57.022969 24745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b5cc1795dd46888c6a20c5ece129f5" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33863 } }
I20260806 22:46:57.023221 24745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:57.023365 24745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:57.023535 24745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:57.026429 24745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b5cc1795dd46888c6a20c5ece129f5" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33863 } }
I20260806 22:46:57.026944 24745 leader_election.cc:302] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79b5cc1795dd46888c6a20c5ece129f5; no voters:
I20260806 22:46:57.027647 24745 leader_election.cc:288] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:57.027753 24757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:57.030352 24745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:57.034759 24757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [term 1 LEADER]: Becoming Leader. State: Replica: 79b5cc1795dd46888c6a20c5ece129f5, State: Running, Role: LEADER
I20260806 22:46:57.035543 24757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b5cc1795dd46888c6a20c5ece129f5" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33863 } }
I20260806 22:46:57.041939 24759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b5cc1795dd46888c6a20c5ece129f5. Latest consensus state: current_term: 1 leader_uuid: "79b5cc1795dd46888c6a20c5ece129f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b5cc1795dd46888c6a20c5ece129f5" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33863 } } }
I20260806 22:46:57.042390 24759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:57.047627 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:57.025720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35361 (local address 127.23.100.190:33863)
0806 22:46:57.025936 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:57.025945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:57.026386 (+ 441us) server_negotiation.cc:407] Connection header received
0806 22:46:57.026628 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:57.026634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:57.026720 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:57.026859 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:57.028956 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:57.030224 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:57.032248 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:57.033152 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:57.033322 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:57.036408 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:57.036415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:57.036422 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:46:57.036466 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:57.042207 (+ 5741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:57.043403 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:57.043413 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:57.043420 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:57.043632 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:46:57.044403 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:46:57.044412 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:46:57.044418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:46:57.044996 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:57.046675 (+ 1679us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:57.046859 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 22:46:57.047162 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 22:46:57.051606 24758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79b5cc1795dd46888c6a20c5ece129f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79b5cc1795dd46888c6a20c5ece129f5" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33863 } } }
I20260806 22:46:57.051952 24758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5 [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:57.055013 24771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:57.060927 24771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:57.070264 24771 catalog_manager.cc:1347] Generated new cluster ID: 956704420cc04fecab8faa803022a2ab
I20260806 22:46:57.070439 24771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:57.107285 24771 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:57.109205 24771 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:57.115821 24771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79b5cc1795dd46888c6a20c5ece129f5: Generated new TSK 0
I20260806 22:46:57.116873 24771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:57.132680 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24658
I20260806 22:46:57.133801 24740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:46:57.233889 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24658
2026-08-06T22:46:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:57 dist-test-slave-zr1t krb5kdc[24788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:57 dist-test-slave-zr1t krb5kdc[24788](info): set up 2 sockets
Aug 06 22:46:57 dist-test-slave-zr1t krb5kdc[24788](info): commencing operation
krb5kdc: starting...
W20260806 22:46:58.576066 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.297s user 0.004s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:46:58 dist-test-slave-zr1t krb5kdc[24788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:46:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:46:58.950260 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:42571
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:46:59.330112 24806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:46:59.330430 24806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:46:59.330536 24806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:46:59.337903 24806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:46:59.338025 24806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:46:59.338084 24806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:46:59.338148 24806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:46:59.338194 24806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:46:59.348178 24806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:42571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:46:59.349581 24806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:46:59.350867 24806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:46:59.363581 24815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:59.363539 24817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:46:59.363636 24814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:46:59.367148 24806 server_base.cc:1034] running on GCE node
I20260806 22:46:59.368069 24806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:46:59.369809 24806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:46:59.371066 24806 hybrid_clock.cc:648] HybridClock initialized: now 1786056419371022 us; error 34 us; skew 500 ppm
I20260806 22:46:59.371479 24806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:46:59 dist-test-slave-zr1t krb5kdc[24788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056419, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:46:59.375293 24806 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:46:59.376983 24806 webserver.cc:460] Webserver started at http://127.23.100.190:39403/ using document root <none> and password file <none>
I20260806 22:46:59.377862 24806 fs_manager.cc:359] Metadata directory not provided
I20260806 22:46:59.378032 24806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:46:59.378386 24806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:46:59.381827 24806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "71b458afc4d949d9a6f26e08d30a5dcc"
format_stamp: "Formatted at 2026-08-06 22:46:59 on dist-test-slave-zr1t"
server_key: "85a54bcfed915f280d8c4ae0d6137a28"
server_key_iv: "3e1d9c11d1677f899c09796d37bc1b05"
server_key_version: "encryptionkey@0"
I20260806 22:46:59.382727 24806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "71b458afc4d949d9a6f26e08d30a5dcc"
format_stamp: "Formatted at 2026-08-06 22:46:59 on dist-test-slave-zr1t"
server_key: "85a54bcfed915f280d8c4ae0d6137a28"
server_key_iv: "3e1d9c11d1677f899c09796d37bc1b05"
server_key_version: "encryptionkey@0"
I20260806 22:46:59.388336 24806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:46:59.391994 24826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:46:59.393356 24806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 22:46:59.393602 24806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "71b458afc4d949d9a6f26e08d30a5dcc"
format_stamp: "Formatted at 2026-08-06 22:46:59 on dist-test-slave-zr1t"
server_key: "85a54bcfed915f280d8c4ae0d6137a28"
server_key_iv: "3e1d9c11d1677f899c09796d37bc1b05"
server_key_version: "encryptionkey@0"
I20260806 22:46:59.393810 24806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:46:59.416550 24806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:46:59.421180 24806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:46:59.421501 24806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:46:59.440210 24806 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:42571
I20260806 22:46:59.440246 24896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:42571 every 8 connection(s)
I20260806 22:46:59.441697 24806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:46:59.445858 24897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:46:59.447594 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24806
I20260806 22:46:59.447783 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:46:59.448233 23954 external_mini_cluster.cc:1428] Setting key af8f61e5c7bb750227a660cafc395002
I20260806 22:46:59.460407 24897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc: Bootstrap starting.
I20260806 22:46:59.461941 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:59.450054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34069 (local address 127.23.100.190:42571)
0806 22:46:59.450938 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:59.450955 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:59.451002 (+ 47us) server_negotiation.cc:407] Connection header received
0806 22:46:59.452642 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:59.452666 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:59.453079 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:59.453425 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:59.454157 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:59.455401 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:59.457400 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:59.458158 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:59.458283 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:59.459603 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:59.459611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:59.459624 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:59.460143 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:59.460217 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:59.460241 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:59.460331 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:59.460627 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 22:46:59.460915 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":456,"threads_started":1}
I20260806 22:46:59.463725 24897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc: Neither blocks nor log segments found. Creating new log.
I20260806 22:46:59.464921 24897 log.cc:824] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc: Log is configured to *not* fsync() on all Append() calls
I20260806 22:46:59.465608 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:46:59.467644 24897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc: No bootstrap required, opened a new log
I20260806 22:46:59.476850 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:46:59.470056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43059 (local address 127.23.100.190:42571)
0806 22:46:59.470250 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 22:46:59.470257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:46:59.470669 (+ 412us) server_negotiation.cc:407] Connection header received
0806 22:46:59.470836 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:46:59.470841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:46:59.470926 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:46:59.471039 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:46:59.471539 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:59.472700 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:59.474099 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:46:59.474832 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:46:59.474972 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:46:59.476074 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:46:59.476080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:46:59.476085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:46:59.476107 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:46:59.476197 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:46:59.476219 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:46:59.476322 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 22:46:59.476436 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 22:46:59.476527 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 22:46:59.477188 24897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b458afc4d949d9a6f26e08d30a5dcc" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42571 } }
I20260806 22:46:59.477720 24897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:46:59.477852 24897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b458afc4d949d9a6f26e08d30a5dcc, State: Initialized, Role: FOLLOWER
I20260806 22:46:59.478505 24897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b458afc4d949d9a6f26e08d30a5dcc" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42571 } }
I20260806 22:46:59.478755 24897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:46:59.478910 24897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:46:59.479092 24897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:46:59.481302 24897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b458afc4d949d9a6f26e08d30a5dcc" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42571 } }
I20260806 22:46:59.481868 24897 leader_election.cc:302] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71b458afc4d949d9a6f26e08d30a5dcc; no voters:
I20260806 22:46:59.482496 24897 leader_election.cc:288] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:46:59.482643 24910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:46:59.483554 24910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [term 1 LEADER]: Becoming Leader. State: Replica: 71b458afc4d949d9a6f26e08d30a5dcc, State: Running, Role: LEADER
I20260806 22:46:59.484184 24910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b458afc4d949d9a6f26e08d30a5dcc" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42571 } }
I20260806 22:46:59.484952 24897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:46:59.487425 24911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71b458afc4d949d9a6f26e08d30a5dcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b458afc4d949d9a6f26e08d30a5dcc" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42571 } } }
I20260806 22:46:59.487833 24911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:59.487530 24914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71b458afc4d949d9a6f26e08d30a5dcc. Latest consensus state: current_term: 1 leader_uuid: "71b458afc4d949d9a6f26e08d30a5dcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b458afc4d949d9a6f26e08d30a5dcc" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 42571 } } }
I20260806 22:46:59.487954 24914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc [sys.catalog]: This master's current role is: LEADER
I20260806 22:46:59.491008 24918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:46:59.496212 24918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:46:59.505903 24918 catalog_manager.cc:1347] Generated new cluster ID: fb7c1a3852c042f996d0678ba9e40de1
I20260806 22:46:59.506048 24918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:46:59.522089 24918 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:46:59.523900 24918 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:46:59.548321 24918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71b458afc4d949d9a6f26e08d30a5dcc: Generated new TSK 0
I20260806 22:46:59.549306 24918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:46:59.559785 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24806
W20260806 22:46:59.561196 24891 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34069
W20260806 22:46:59.562124 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24806: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:46:59.562326 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24806
2026-08-06T22:46:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:46:59 dist-test-slave-zr1t krb5kdc[24937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:46:59 dist-test-slave-zr1t krb5kdc[24937](info): set up 2 sockets
Aug 06 22:46:59 dist-test-slave-zr1t krb5kdc[24937](info): commencing operation
krb5kdc: starting...
W20260806 22:47:00.915859 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.309s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:47:01 dist-test-slave-zr1t krb5kdc[24937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:47:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:47:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:47:01.278064 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33599
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:33599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:47:01.622154 24952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:47:01.622468 24952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:47:01.622632 24952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:47:01.630021 24952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:47:01.630127 24952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:47:01.630265 24952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:47:01.630379 24952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:47:01.630458 24952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:47:01.640450 24952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:33599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:33599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:47:01.642004 24952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:47:01.643481 24952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:47:01.655455 24959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:01.655632 24962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:01.655642 24960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:47:01.656544 24952 server_base.cc:1034] running on GCE node
I20260806 22:47:01.657356 24952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:47:01.658751 24952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:47:01.660003 24952 hybrid_clock.cc:648] HybridClock initialized: now 1786056421659964 us; error 57 us; skew 500 ppm
I20260806 22:47:01.660404 24952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:47:01 dist-test-slave-zr1t krb5kdc[24937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056421, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:47:01.663738 24952 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:47:01.665432 24952 webserver.cc:460] Webserver started at http://127.23.100.190:36191/ using document root <none> and password file <none>
I20260806 22:47:01.666179 24952 fs_manager.cc:359] Metadata directory not provided
I20260806 22:47:01.666352 24952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:47:01.666700 24952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:47:01.670006 24952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "6709131a18d14c1f99490bb314a97756"
format_stamp: "Formatted at 2026-08-06 22:47:01 on dist-test-slave-zr1t"
server_key: "264f1c88b5f9dc59c7ffc63af64d058c"
server_key_iv: "e7e7b78a0abf7788452eb468da98231a"
server_key_version: "encryptionkey@0"
I20260806 22:47:01.670861 24952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "6709131a18d14c1f99490bb314a97756"
format_stamp: "Formatted at 2026-08-06 22:47:01 on dist-test-slave-zr1t"
server_key: "264f1c88b5f9dc59c7ffc63af64d058c"
server_key_iv: "e7e7b78a0abf7788452eb468da98231a"
server_key_version: "encryptionkey@0"
I20260806 22:47:01.676463 24952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 22:47:01.680160 24974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:47:01.681480 24952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 22:47:01.681725 24952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "6709131a18d14c1f99490bb314a97756"
format_stamp: "Formatted at 2026-08-06 22:47:01 on dist-test-slave-zr1t"
server_key: "264f1c88b5f9dc59c7ffc63af64d058c"
server_key_iv: "e7e7b78a0abf7788452eb468da98231a"
server_key_version: "encryptionkey@0"
I20260806 22:47:01.681954 24952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:47:01.687722 24952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:47:01.692145 24952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:47:01.692459 24952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:47:01.711449 24952 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:33599
I20260806 22:47:01.711489 25045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:33599 every 8 connection(s)
I20260806 22:47:01.712821 24952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:47:01.716454 25046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:47:01.721609 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 24952
I20260806 22:47:01.721807 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:47:01.722391 23954 external_mini_cluster.cc:1428] Setting key 0c6536a29fd3f673edd5ec10dc672fa6
I20260806 22:47:01.728554 25046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756: Bootstrap starting.
I20260806 22:47:01.730587 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:01.724251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43565 (local address 127.23.100.190:33599)
0806 22:47:01.725014 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:01.725028 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:01.725067 (+ 39us) server_negotiation.cc:407] Connection header received
0806 22:47:01.727238 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:01.727259 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:01.727566 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:01.727948 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:01.728992 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:01.728999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:01.729010 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:47:01.729029 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:01.729087 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:47:01.729106 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:01.729340 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:01.729480 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 22:47:01.729693 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":340,"threads_started":1}
I20260806 22:47:01.732070 25046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756: Neither blocks nor log segments found. Creating new log.
I20260806 22:47:01.733270 25046 log.cc:824] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756: Log is configured to *not* fsync() on all Append() calls
I20260806 22:47:01.734459 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:47:01.736497 25046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756: No bootstrap required, opened a new log
I20260806 22:47:01.742347 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:01.739737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47971 (local address 127.23.100.190:33599)
0806 22:47:01.739930 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:01.739938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:01.740358 (+ 420us) server_negotiation.cc:407] Connection header received
0806 22:47:01.740505 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:01.740509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:01.740572 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:01.740683 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:01.741635 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:01.741640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:01.741645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 22:47:01.741662 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:01.741711 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 22:47:01.741733 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:01.741872 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:01.741984 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 22:47:01.742050 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 22:47:01.745563 25046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709131a18d14c1f99490bb314a97756" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33599 } }
I20260806 22:47:01.745937 25046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 22:47:01.746044 25046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6709131a18d14c1f99490bb314a97756, State: Initialized, Role: FOLLOWER
I20260806 22:47:01.746712 25046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709131a18d14c1f99490bb314a97756" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33599 } }
I20260806 22:47:01.746999 25046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 22:47:01.747108 25046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 22:47:01.747234 25046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:47:01.749641 25046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709131a18d14c1f99490bb314a97756" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33599 } }
I20260806 22:47:01.750159 25046 leader_election.cc:302] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6709131a18d14c1f99490bb314a97756; no voters:
I20260806 22:47:01.750764 25046 leader_election.cc:288] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 22:47:01.750928 25059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:47:01.751613 25059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [term 1 LEADER]: Becoming Leader. State: Replica: 6709131a18d14c1f99490bb314a97756, State: Running, Role: LEADER
I20260806 22:47:01.752170 25059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709131a18d14c1f99490bb314a97756" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33599 } }
I20260806 22:47:01.752826 25046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:47:01.755219 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6709131a18d14c1f99490bb314a97756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709131a18d14c1f99490bb314a97756" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33599 } } }
I20260806 22:47:01.755156 25062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6709131a18d14c1f99490bb314a97756. Latest consensus state: current_term: 1 leader_uuid: "6709131a18d14c1f99490bb314a97756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709131a18d14c1f99490bb314a97756" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 33599 } } }
I20260806 22:47:01.755685 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [sys.catalog]: This master's current role is: LEADER
I20260806 22:47:01.755710 25062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756 [sys.catalog]: This master's current role is: LEADER
I20260806 22:47:01.757922 25067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:47:01.763305 25067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:47:01.772290 25067 catalog_manager.cc:1347] Generated new cluster ID: d0c6cbccafaa4a20852f8640b11be8b0
I20260806 22:47:01.772419 25067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:47:01.790762 25067 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:47:01.791275 25067 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:47:01.798130 25067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6709131a18d14c1f99490bb314a97756: Generated new TSK 0
I20260806 22:47:01.799111 25067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:47:01.806200 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24952
W20260806 22:47:01.807432 25039 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43565
W20260806 22:47:01.808183 23954 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 24952: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 22:47:01.808342 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 24952
2026-08-06T22:47:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2248 ms)
[----------] 18 tests from AuthTokenIssuingTest (41599 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:47:01 dist-test-slave-zr1t krb5kdc[25087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:47:01 dist-test-slave-zr1t krb5kdc[25087](info): set up 2 sockets
Aug 06 22:47:01 dist-test-slave-zr1t krb5kdc[25087](info): commencing operation
krb5kdc: starting...
W20260806 22:47:03.148394 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:47:03 dist-test-slave-zr1t krb5kdc[25087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:47:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:47:03.519495 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:43117
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:43117,127.23.100.189:43801,127.23.100.188:44515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:47:03.879571 25106 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 22:47:03.880065 25106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:47:03.880175 25106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:47:03.880334 25106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:47:03.888605 25106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:47:03.888707 25106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:47:03.888815 25106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:47:03.888918 25106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:47:03.889027 25106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:47:03.899446 25106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:43117,127.23.100.189:43801,127.23.100.188:44515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:47:03.900849 25106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:47:03.902309 25106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:47:03.914639 25114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:03.914673 25115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:03.914731 25118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:47:03.916177 25106 server_base.cc:1034] running on GCE node
I20260806 22:47:03.917220 25106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:47:03.918769 25106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:47:03.919994 25106 hybrid_clock.cc:648] HybridClock initialized: now 1786056423919928 us; error 56 us; skew 500 ppm
I20260806 22:47:03.920357 25106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:47:03 dist-test-slave-zr1t krb5kdc[25087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056423, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:47:03.923629 25106 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:47:03.925076 25106 webserver.cc:460] Webserver started at http://127.23.100.190:38655/ using document root <none> and password file <none>
I20260806 22:47:03.925881 25106 fs_manager.cc:359] Metadata directory not provided
I20260806 22:47:03.926012 25106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:47:03.926360 25106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:47:03.929618 25106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "eb8ee3386bb04c76a83bb5f3124088a6"
format_stamp: "Formatted at 2026-08-06 22:47:03 on dist-test-slave-zr1t"
server_key: "e152903415f892979fd72c876e9d2134"
server_key_iv: "0c31ba46adb36b5aad79ee4a1a63e977"
server_key_version: "encryptionkey@0"
I20260806 22:47:03.930552 25106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "eb8ee3386bb04c76a83bb5f3124088a6"
format_stamp: "Formatted at 2026-08-06 22:47:03 on dist-test-slave-zr1t"
server_key: "e152903415f892979fd72c876e9d2134"
server_key_iv: "0c31ba46adb36b5aad79ee4a1a63e977"
server_key_version: "encryptionkey@0"
I20260806 22:47:03.936327 25106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 22:47:03.940135 25128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:47:03.941847 25106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 22:47:03.942096 25106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "eb8ee3386bb04c76a83bb5f3124088a6"
format_stamp: "Formatted at 2026-08-06 22:47:03 on dist-test-slave-zr1t"
server_key: "e152903415f892979fd72c876e9d2134"
server_key_iv: "0c31ba46adb36b5aad79ee4a1a63e977"
server_key_version: "encryptionkey@0"
I20260806 22:47:03.942389 25106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:47:03.960500 25106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:47:03.965189 25106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:47:03.965636 25106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:47:03.990952 25106 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:43117
I20260806 22:47:03.990993 25203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:43117 every 8 connection(s)
I20260806 22:47:03.992394 25106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:47:03.996490 25204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:47:03.998680 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 25106
I20260806 22:47:03.998958 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:47:03.999631 23954 external_mini_cluster.cc:1428] Setting key cb78ba1e3fd2b8bdb5fd06ad44b70b1e
I20260806 22:47:04.004144 25204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:47:04 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056423, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:47:04.031227 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.007732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.23.100.190:43117)
0806 22:47:04.009703 (+ 1971us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:04.009712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:04.009767 (+ 55us) server_negotiation.cc:407] Connection header received
0806 22:47:04.011574 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:04.011580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:04.011843 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:04.012085 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:04.013598 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.014954 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.017325 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.018068 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.018300 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:04.022488 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:04.022499 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:04.022513 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:04.022551 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:04.026015 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:04.026865 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:04.026875 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:04.026877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:04.027058 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:04.027505 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:04.027513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:04.027516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:04.028099 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:04.028337 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:04.028925 (+ 588us) server_negotiation.cc:299] Negotiation successful
0806 22:47:04.029443 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1668,"spinlock_wait_cycles":15616,"thread_start_us":1424,"threads_started":1}
I20260806 22:47:04.031258 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.007731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:43117 (local address 127.0.0.1:47832)
0806 22:47:04.008735 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.008786 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:04.011129 (+ 2343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:04.012099 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:04.012107 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:04.012941 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:04.013315 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:04.013333 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.015148 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.015158 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:04.017152 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:04.017169 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.018341 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.018349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:04.018543 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:04.019311 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:04.019339 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:04.022158 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:04.026182 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:04.026197 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:04.026213 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:04.026690 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:04.027196 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:04.027201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:04.027205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:04.027382 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:04.028268 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:04.028283 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:04.028526 (+ 243us) client_negotiation.cc:769] Sending connection context
0806 22:47:04.028810 (+ 284us) client_negotiation.cc:240] Negotiation successful
0806 22:47:04.029609 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":267,"thread_start_us":458,"threads_started":1}
WARNING: no policy specified for kudu/127.23.100.189@KRBTEST.COM; defaulting to no policy
I20260806 22:47:04.035010 25204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } has no permanent_uuid. Determining permanent_uuid...
I20260806 22:47:04.036242 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.035802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55678)
0806 22:47:04.036001 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.036117 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260806 22:47:04.036600 25132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.189@KRBTEST.COM" created.
I20260806 22:47:04.038457 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.037981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55690)
0806 22:47:04.038180 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.038281 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 22:47:04.038697 25204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:43801: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab.
Entry for principal kudu/127.23.100.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab.
I20260806 22:47:04.079028 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.189:43801
--webserver_interface=127.23.100.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab
--principal=kudu/127.23.100.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:43117,127.23.100.189:43801,127.23.100.188:44515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:47:04.088121 25204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } attempt: 1
I20260806 22:47:04.089704 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.088951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55702)
0806 22:47:04.089223 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.089434 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 22:47:04.090999 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.090478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55704)
0806 22:47:04.090687 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.090800 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 22:47:04.091264 25204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:43801: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
I20260806 22:47:04.159552 25204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } attempt: 2
I20260806 22:47:04.160981 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.160330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55720)
0806 22:47:04.160736 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.160842 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233}
I20260806 22:47:04.162163 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.161739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55722)
0806 22:47:04.161932 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.162018 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 22:47:04.162365 25204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:43801: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
I20260806 22:47:04.253657 25204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } attempt: 3
I20260806 22:47:04.254942 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.254453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55738)
0806 22:47:04.254672 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.254778 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 22:47:04.256106 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.255699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55750)
0806 22:47:04.255886 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.255959 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 22:47:04.256318 25204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:43801: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
I20260806 22:47:04.399622 25204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } attempt: 4
I20260806 22:47:04.400895 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.400341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55764)
0806 22:47:04.400672 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.400761 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260806 22:47:04.402107 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.401714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55778)
0806 22:47:04.401889 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.401967 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260806 22:47:04.402479 25204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:43801: Network error: Client connection negotiation failed: client connection to 127.23.100.189:43801: connect: Connection refused (error 111)
W20260806 22:47:04.434970 25216 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 22:47:04.435333 25216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:47:04.435462 25216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:47:04.435601 25216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:47:04.442919 25216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:47:04.443014 25216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:47:04.443102 25216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:47:04.443179 25216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:47:04.443272 25216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:47:04.453419 25216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:43117,127.23.100.189:43801,127.23.100.188:44515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab
--principal=kudu/127.23.100.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.189:43801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.100.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:47:04.454842 25216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:47:04.456202 25216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:47:04.468230 25222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:04.468402 25224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:04.468428 25221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:47:04.469383 25216 server_base.cc:1034] running on GCE node
I20260806 22:47:04.470280 25216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:47:04.471674 25216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:47:04.472929 25216 hybrid_clock.cc:648] HybridClock initialized: now 1786056424472846 us; error 61 us; skew 500 ppm
I20260806 22:47:04.473258 25216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:47:04 dist-test-slave-zr1t krb5kdc[25087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056424, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:47:04.476639 25216 init.cc:377] Logged in from keytab as kudu/127.23.100.189@KRBTEST.COM (short username kudu)
I20260806 22:47:04.478484 25216 webserver.cc:460] Webserver started at http://127.23.100.189:34417/ using document root <none> and password file <none>
I20260806 22:47:04.479372 25216 fs_manager.cc:359] Metadata directory not provided
I20260806 22:47:04.479514 25216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:47:04.479914 25216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:47:04.483027 25216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/data/instance:
uuid: "a9331eefc6d04313888f7f326279bd53"
format_stamp: "Formatted at 2026-08-06 22:47:04 on dist-test-slave-zr1t"
server_key: "69878cf463bfb25cda06ddd14d0f1c30"
server_key_iv: "a312627755402533d3255c80e87b73bc"
server_key_version: "encryptionkey@0"
I20260806 22:47:04.483795 25216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/wal/instance:
uuid: "a9331eefc6d04313888f7f326279bd53"
format_stamp: "Formatted at 2026-08-06 22:47:04 on dist-test-slave-zr1t"
server_key: "69878cf463bfb25cda06ddd14d0f1c30"
server_key_iv: "a312627755402533d3255c80e87b73bc"
server_key_version: "encryptionkey@0"
I20260806 22:47:04.489204 25216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 22:47:04.492905 25235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:47:04.494488 25216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 22:47:04.494648 25216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/wal
uuid: "a9331eefc6d04313888f7f326279bd53"
format_stamp: "Formatted at 2026-08-06 22:47:04 on dist-test-slave-zr1t"
server_key: "69878cf463bfb25cda06ddd14d0f1c30"
server_key_iv: "a312627755402533d3255c80e87b73bc"
server_key_version: "encryptionkey@0"
I20260806 22:47:04.494844 25216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:47:04.507622 25216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:47:04.512333 25216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:47:04.512581 25216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:47:04.531833 25216 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.189:43801
I20260806 22:47:04.531890 25305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.189:43801 every 8 connection(s)
I20260806 22:47:04.533335 25216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
I20260806 22:47:04.537171 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 25216
I20260806 22:47:04.537204 25306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:47:04.537413 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-1/wal/instance
I20260806 22:47:04.537969 23954 external_mini_cluster.cc:1428] Setting key 43ada6de49959876f02cf7fb6725361a
I20260806 22:47:04.544059 25306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:47:04 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056424, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:47:04.567219 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.546898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.23.100.190:43117)
0806 22:47:04.547420 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:04.547425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:04.548584 (+ 1159us) server_negotiation.cc:407] Connection header received
0806 22:47:04.550629 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:04.550634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:04.550730 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:04.550857 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:04.552461 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.553168 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.555451 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.555930 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.556053 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:04.560589 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:04.560595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:04.560598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:04.560646 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:04.563669 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:04.564540 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:04.564545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:04.564547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:04.564681 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:04.565164 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:04.565168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:04.565170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:04.565495 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:04.565720 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:04.566559 (+ 839us) server_negotiation.cc:299] Negotiation successful
0806 22:47:04.566833 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":302,"threads_started":1}
I20260806 22:47:04.568269 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.547555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:43117 (local address 127.0.0.1:47836)
0806 22:47:04.548407 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.548460 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:04.550342 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:04.551107 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:04.551115 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:04.551640 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:04.552213 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:04.552230 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.553352 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.553356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:04.555251 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:04.555268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.556186 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.556195 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:04.556535 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:04.557339 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:04.557377 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:04.560312 (+ 2935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:04.563850 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:04.563860 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:04.563889 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:04.564380 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:04.564805 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:04.564812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:04.564817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:04.565026 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:04.565710 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:04.565718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:04.566129 (+ 411us) client_negotiation.cc:769] Sending connection context
0806 22:47:04.566438 (+ 309us) client_negotiation.cc:240] Negotiation successful
0806 22:47:04.567042 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":363,"threads_started":1}
I20260806 22:47:04.570282 25306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.100.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.188@KRBTEST.COM" created.
Aug 06 22:47:04 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056424, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
I20260806 22:47:04.585693 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.571229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55784)
0806 22:47:04.571445 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.571466 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:04.571585 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:04.572366 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:04.572372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:04.572884 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:04.573134 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:04.573150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.574635 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.574639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:04.575478 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:04.575489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.576399 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.576404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:04.576611 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:04.577229 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:04.577257 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:04.579863 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:04.583145 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:04.583153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:04.583159 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:04.583533 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:04.583896 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:04.583902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:04.583905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:04.584049 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:04.584638 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:04.584644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:04.584783 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 22:47:04.584923 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 22:47:04.585217 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":3}
I20260806 22:47:04.585757 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.571277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.23.100.189:43801)
0806 22:47:04.571929 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:04.571935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:04.571966 (+ 31us) server_negotiation.cc:407] Connection header received
0806 22:47:04.572011 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:04.572017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:04.572193 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:04.572307 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:04.573319 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.574517 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.575711 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.576245 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.576398 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:04.580095 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:04.580107 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:04.580121 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:04.580149 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:04.583010 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:04.583665 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:04.583681 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:04.583683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:04.583789 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:04.584149 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:04.584156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:04.584159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:04.584518 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:04.584711 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:04.585079 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 22:47:04.585326 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":500,"thread_start_us":330,"threads_started":1}
I20260806 22:47:04.588527 25306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } has no permanent_uuid. Determining permanent_uuid...
I20260806 22:47:04.589757 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.589221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36306)
0806 22:47:04.589429 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.589596 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260806 22:47:04.590005 25240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
I20260806 22:47:04.591619 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.591247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36308)
0806 22:47:04.591431 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.591511 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 22:47:04.591858 25306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:44515: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab.
Entry for principal kudu/127.23.100.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab.
I20260806 22:47:04.618407 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.188:44515
--webserver_interface=127.23.100.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab
--principal=kudu/127.23.100.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:43117,127.23.100.189:43801,127.23.100.188:44515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:47:04.641207 25306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } attempt: 1
I20260806 22:47:04.642722 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.642033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36310)
0806 22:47:04.642339 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.642498 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 22:47:04.643994 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.643529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36314)
0806 22:47:04.643728 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.643838 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260806 22:47:04.644277 25306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:44515: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
I20260806 22:47:04.701789 25204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } attempt: 5
I20260806 22:47:04.712529 25306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } attempt: 2
Aug 06 22:47:04 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056423, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
I20260806 22:47:04.713595 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.713114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36318)
0806 22:47:04.713292 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.713390 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 22:47:04.714789 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.714328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36328)
0806 22:47:04.714507 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.714599 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 22:47:04.715221 25306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:44515: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
I20260806 22:47:04.720121 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.702809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.23.100.189:43801)
0806 22:47:04.703037 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:04.703046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:04.703069 (+ 23us) server_negotiation.cc:407] Connection header received
0806 22:47:04.703140 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:04.703145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:04.703206 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:04.703318 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:04.704122 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.704864 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.706163 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.706632 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.706782 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:04.714093 (+ 7311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:04.714101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:04.714106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:04.714146 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:04.717183 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:04.718085 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:04.718089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:04.718091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:04.718212 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:04.718681 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:04.718686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:04.718689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:04.719075 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:04.719195 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:04.719760 (+ 565us) server_negotiation.cc:299] Negotiation successful
0806 22:47:04.719935 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 22:47:04.720494 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.702583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55792)
0806 22:47:04.702796 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.702824 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:04.702977 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:04.703362 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:04.703368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:04.703796 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:04.703958 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:04.703966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.705008 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.705016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:04.706004 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:04.706017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:04.706789 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:04.706795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:04.707013 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:04.711328 (+ 4315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:04.711348 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:04.713845 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:04.717358 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:04.717377 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:04.717383 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:04.717958 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:04.718326 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:04.718331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:04.718334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:04.718515 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:04.719198 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:04.719203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:04.719428 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 22:47:04.719612 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 22:47:04.720020 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 22:47:04.721602 25204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } has no permanent_uuid. Determining permanent_uuid...
I20260806 22:47:04.722651 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.722203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36344)
0806 22:47:04.722399 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.722495 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 22:47:04.723675 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.723292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36350)
0806 22:47:04.723463 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.723546 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 22:47:04.723913 25204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:44515: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
I20260806 22:47:04.775204 25204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } attempt: 1
I20260806 22:47:04.776647 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.776047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36364)
0806 22:47:04.776302 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.776436 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260806 22:47:04.777777 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.777348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36372)
0806 22:47:04.777509 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.777630 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 22:47:04.778064 25204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:44515: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
I20260806 22:47:04.806550 25306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } attempt: 3
I20260806 22:47:04.807706 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.807249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36382)
0806 22:47:04.807443 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.807547 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 22:47:04.808919 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.808499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36392)
0806 22:47:04.808730 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.808809 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260806 22:47:04.809214 25306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:44515: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
I20260806 22:47:04.846437 25204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } attempt: 2
I20260806 22:47:04.847862 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.847212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36404)
0806 22:47:04.847503 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.847617 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 22:47:04.848971 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.848607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36420)
0806 22:47:04.848776 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.848851 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260806 22:47:04.849349 25204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:44515: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
I20260806 22:47:04.952454 25306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } attempt: 4
I20260806 22:47:04.953889 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.953228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36436)
0806 22:47:04.953534 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.953669 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 22:47:04.955220 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.954737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36440)
0806 22:47:04.954940 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.955038 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 22:47:04.955552 25306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:44515: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
I20260806 22:47:04.955623 25204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } attempt: 3
I20260806 22:47:04.956805 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.956247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36456)
0806 22:47:04.956477 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.956593 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 22:47:04.958060 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:04.957513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36472)
0806 22:47:04.957787 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:04.957909 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260806 22:47:04.958237 25204 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:44515: Network error: Client connection negotiation failed: client connection to 127.23.100.188:44515: connect: Connection refused (error 111)
W20260806 22:47:04.986140 25314 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 22:47:04.986552 25314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:47:04.986685 25314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:47:04.986824 25314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:47:04.996778 25314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:47:04.996891 25314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:47:04.996966 25314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:47:04.997035 25314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:47:04.997115 25314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:47:05.010082 25314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:43117,127.23.100.189:43801,127.23.100.188:44515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab
--principal=kudu/127.23.100.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.188:44515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.100.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:47:05.011816 25314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:47:05.013507 25314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:47:05.026607 25326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:05.026552 25321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:05.027312 25323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:47:05.027832 25314 server_base.cc:1034] running on GCE node
I20260806 22:47:05.028852 25314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:47:05.030876 25314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:47:05.032156 25314 hybrid_clock.cc:648] HybridClock initialized: now 1786056425032076 us; error 40 us; skew 500 ppm
I20260806 22:47:05.032647 25314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:47:05 dist-test-slave-zr1t krb5kdc[25087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:47:05.037014 25314 init.cc:377] Logged in from keytab as kudu/127.23.100.188@KRBTEST.COM (short username kudu)
I20260806 22:47:05.038623 25314 webserver.cc:460] Webserver started at http://127.23.100.188:39291/ using document root <none> and password file <none>
I20260806 22:47:05.039366 25314 fs_manager.cc:359] Metadata directory not provided
I20260806 22:47:05.039513 25314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:47:05.039855 25314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:47:05.043280 25314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/data/instance:
uuid: "f5285a0a620a446d9c946c6b515084ff"
format_stamp: "Formatted at 2026-08-06 22:47:05 on dist-test-slave-zr1t"
server_key: "4374da5feb5211c7f130382a460e4ba0"
server_key_iv: "047d94b86beab27bf5308edd9bc2f792"
server_key_version: "encryptionkey@0"
I20260806 22:47:05.044517 25314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/wal/instance:
uuid: "f5285a0a620a446d9c946c6b515084ff"
format_stamp: "Formatted at 2026-08-06 22:47:05 on dist-test-slave-zr1t"
server_key: "4374da5feb5211c7f130382a460e4ba0"
server_key_iv: "047d94b86beab27bf5308edd9bc2f792"
server_key_version: "encryptionkey@0"
I20260806 22:47:05.052493 25314 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260806 22:47:05.057063 25340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:47:05.058677 25314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:47:05.058890 25314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/wal
uuid: "f5285a0a620a446d9c946c6b515084ff"
format_stamp: "Formatted at 2026-08-06 22:47:05 on dist-test-slave-zr1t"
server_key: "4374da5feb5211c7f130382a460e4ba0"
server_key_iv: "047d94b86beab27bf5308edd9bc2f792"
server_key_version: "encryptionkey@0"
I20260806 22:47:05.059140 25314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:47:05.096760 25314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:47:05.101495 25314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:47:05.101892 25314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:47:05.122838 25314 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.188:44515
I20260806 22:47:05.122934 25407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.188:44515 every 8 connection(s)
I20260806 22:47:05.124320 25314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
I20260806 22:47:05.124657 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 25314
I20260806 22:47:05.124867 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786056046488940-23954-0/minicluster-data/master-2/wal/instance
I20260806 22:47:05.125363 23954 external_mini_cluster.cc:1428] Setting key 695ef075c1783beddb1a12006c24618a
I20260806 22:47:05.131070 25408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:47:05.135478 25204 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } attempt: 4
I20260806 22:47:05.139449 25408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:47:05 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
Aug 06 22:47:05 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056423, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
I20260806 22:47:05.154899 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.136168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36486)
0806 22:47:05.136494 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:05.136516 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:05.136644 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:05.137764 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:05.137773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:05.138351 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:05.138555 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:05.138566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.139922 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.139929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:05.141038 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:05.141059 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.142091 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.142100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:05.142350 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.143087 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:05.143121 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:05.147015 (+ 3894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:05.151914 (+ 4899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:05.151920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:05.151924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:05.152322 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:05.152738 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:05.152742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:05.152746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:05.152929 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:05.153810 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:05.153815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:05.154005 (+ 190us) client_negotiation.cc:769] Sending connection context
0806 22:47:05.154182 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 22:47:05.154509 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260806 22:47:05.156595 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.136365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.23.100.188:44515)
0806 22:47:05.137336 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.137341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.137365 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:47:05.137412 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.137417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.137488 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.137722 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.138774 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.139748 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.141204 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.141904 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.142040 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.147389 (+ 5349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:05.147398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:05.147406 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:05.147461 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:05.151784 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.152453 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.152459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.152463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.152612 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.153059 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.153064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.153073 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.153664 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:05.153849 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:05.154463 (+ 614us) server_negotiation.cc:299] Negotiation successful
0806 22:47:05.154944 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":782,"thread_start_us":517,"threads_started":1}
I20260806 22:47:05.156613 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.127498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.23.100.188:44515)
0806 22:47:05.128223 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.128235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.128307 (+ 72us) server_negotiation.cc:407] Connection header received
0806 22:47:05.130261 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.130291 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.130669 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.131063 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.132782 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.134340 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.135831 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.137182 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.137389 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.143510 (+ 6121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:05.143527 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:05.143553 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:05.143617 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:05.150889 (+ 7272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.152026 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.152036 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.152047 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.152344 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.152945 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.152950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.152953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.153702 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:05.155160 (+ 1458us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:05.155344 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 22:47:05.155572 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":519,"spinlock_wait_cycles":19840,"thread_start_us":307,"threads_started":1}
Aug 06 22:47:05 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:47:05.165341 25204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6: Bootstrap starting.
I20260806 22:47:05.169750 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.142778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:43117 (local address 127.0.0.1:47842)
0806 22:47:05.149853 (+ 7075us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:05.149926 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:05.150120 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:05.151070 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:05.151081 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:05.152143 (+ 1062us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:05.152581 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:05.152599 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.154351 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.154357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:05.156333 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:05.156349 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.157464 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.157471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:05.157712 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.158475 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:05.158494 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:05.161621 (+ 3127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:05.166225 (+ 4604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:05.166237 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:05.166243 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:05.166825 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:05.167318 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:05.167324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:05.167330 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:05.167543 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:05.168388 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:05.168398 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:05.168598 (+ 200us) client_negotiation.cc:769] Sending connection context
0806 22:47:05.168850 (+ 252us) client_negotiation.cc:240] Negotiation successful
0806 22:47:05.169250 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6797,"thread_start_us":5819,"threads_started":1}
I20260806 22:47:05.169868 25204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6: Neither blocks nor log segments found. Creating new log.
I20260806 22:47:05.169991 25415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.142800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.23.100.190:43117)
0806 22:47:05.150685 (+ 7885us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.150693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.150713 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:47:05.150784 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.150790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.150868 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.150987 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.152795 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.154097 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.156568 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.157232 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.157372 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.161847 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:05.161854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:05.161865 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:05.161903 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:05.166048 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.166971 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.166976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.166979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.167137 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.167686 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.167691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.167695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.168206 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:05.168403 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:05.169002 (+ 599us) server_negotiation.cc:299] Negotiation successful
0806 22:47:05.169320 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7724,"thread_start_us":7573,"threads_started":1}
Aug 06 22:47:05 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:47:05.171458 25204 log.cc:824] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6: Log is configured to *not* fsync() on all Append() calls
I20260806 22:47:05.172159 25408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } has no permanent_uuid. Determining permanent_uuid...
I20260806 22:47:05.175487 25204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6: No bootstrap required, opened a new log
Aug 06 22:47:05 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
I20260806 22:47:05.182610 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.160215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.23.100.190:43117)
0806 22:47:05.160930 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.160936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.160960 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:47:05.161012 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.161017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.161092 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.161212 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.163108 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.164545 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.166129 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.166649 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.166799 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.172713 (+ 5914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:05.172720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:05.172728 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:05.172776 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:05.178248 (+ 5472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.179226 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.179232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.179236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.179468 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.180023 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.180030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.180033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.180499 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:05.180817 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:05.181757 (+ 940us) server_negotiation.cc:299] Negotiation successful
0806 22:47:05.182269 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":321,"threads_started":1}
I20260806 22:47:05.185711 25204 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } }
I20260806 22:47:05.186282 25204 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8ee3386bb04c76a83bb5f3124088a6, State: Initialized, Role: FOLLOWER
I20260806 22:47:05.186870 25204 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } }
I20260806 22:47:05.188005 25421 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } } }
I20260806 22:47:05.188484 25421 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:05.189599 25204 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:47:05.190595 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.173148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.23.100.189:43801)
0806 22:47:05.173384 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.173391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.173835 (+ 444us) server_negotiation.cc:407] Connection header received
0806 22:47:05.173905 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.173912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.173990 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.174086 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.175178 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.176022 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.178084 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.178705 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.178789 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.183122 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:05.183132 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:05.183137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:05.183184 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:05.187021 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.188040 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.188046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.188050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.188261 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.188841 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.188849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.188853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.189222 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:05.189376 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:05.189967 (+ 591us) server_negotiation.cc:299] Negotiation successful
0806 22:47:05.190219 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 22:47:05.190593 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.173157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:43801 (local address 127.0.0.1:55798)
0806 22:47:05.173572 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:05.173593 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:05.173715 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:05.174161 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:05.174168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:05.174761 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:05.175012 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:05.175023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.176186 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.176193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:05.177878 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:05.177898 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.179025 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.179029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:05.179221 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.179915 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:05.179945 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:05.182838 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:05.187185 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:05.187191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:05.187195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:05.187886 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:05.188427 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:05.188432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:05.188435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:05.188630 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:05.189369 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:05.189376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:05.189609 (+ 233us) client_negotiation.cc:769] Sending connection context
0806 22:47:05.189824 (+ 215us) client_negotiation.cc:240] Negotiation successful
0806 22:47:05.190150 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259}
I20260806 22:47:05.192138 25408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:47:05 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
Aug 06 22:47:05 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
I20260806 22:47:05.205118 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.184655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.23.100.189:43801)
0806 22:47:05.185409 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.185416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.185442 (+ 26us) server_negotiation.cc:407] Connection header received
0806 22:47:05.185572 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.185578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.185656 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.185794 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.187456 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.188823 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.190753 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.191471 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.191610 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.197198 (+ 5588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:05.197206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:05.197211 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:05.197253 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:05.201091 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.202102 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.202110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.202114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.202296 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.202981 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.202987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.202989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.203390 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:05.204541 (+ 1151us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:05.204641 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 22:47:05.204820 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":338,"threads_started":1}
I20260806 22:47:05.211751 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.193741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.23.100.188:44515)
0806 22:47:05.193986 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.193994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.194020 (+ 26us) server_negotiation.cc:407] Connection header received
0806 22:47:05.194088 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.194096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.194188 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.194311 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.195434 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.196858 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.198483 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.199453 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.199618 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.203864 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:05.203874 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:05.203881 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:05.203931 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:05.207843 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.208565 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.208574 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.208580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.209225 (+ 645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.209762 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.209767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.209770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.210089 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:05.210477 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:05.211115 (+ 638us) server_negotiation.cc:299] Negotiation successful
0806 22:47:05.211322 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":83}
I20260806 22:47:05.212227 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:47:05.212596 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.193448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36502)
0806 22:47:05.193834 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:05.193856 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:05.193979 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:05.194358 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:05.194364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:05.195023 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:05.195260 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:05.195275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.197031 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.197040 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:05.198287 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:05.198304 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.199632 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.199639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:05.199898 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.200567 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:05.200594 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:05.203624 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:05.207980 (+ 4356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:05.207987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:05.207991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:05.208430 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:05.209386 (+ 956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:05.209393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:05.209398 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:05.209636 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:05.210562 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:05.210570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:05.210768 (+ 198us) client_negotiation.cc:769] Sending connection context
0806 22:47:05.211960 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0806 22:47:05.212270 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":26}
I20260806 22:47:05.215536 25164 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "eb8ee3386bb04c76a83bb5f3124088a6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47844
I20260806 22:47:05.215987 25164 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 22:47:05.216213 25164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [term 0 FOLLOWER]: Advancing to term 1
W20260806 22:47:05.216588 25440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 22:47:05.216733 25440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 22:47:05.219456 25135 tablet.cc:2369] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:47:05.221490 25408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff: Bootstrap starting.
I20260806 22:47:05.222728 25164 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } }
I20260806 22:47:05.224602 25164 leader_election.cc:288] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [CANDIDATE]: Term 1 election: Requested vote from peers a9331eefc6d04313888f7f326279bd53 (127.23.100.189:43801), f5285a0a620a446d9c946c6b515084ff (127.23.100.188:44515)
I20260806 22:47:05.226532 25408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff: Neither blocks nor log segments found. Creating new log.
I20260806 22:47:05.227969 25408 log.cc:824] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff: Log is configured to *not* fsync() on all Append() calls
I20260806 22:47:05.227623 25106 master_runner.cc:186] Error getting master registration for 127.23.100.189:43801: OK, instance_id { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" instance_seqno: 1786056424526451 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 22:47:05.241250 25106 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 22:47:05.245250 25408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff: No bootstrap required, opened a new log
I20260806 22:47:05.254765 25306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } attempt: 5
I20260806 22:47:05.258864 25408 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } }
I20260806 22:47:05.259402 25408 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5285a0a620a446d9c946c6b515084ff, State: Initialized, Role: FOLLOWER
I20260806 22:47:05.259971 25408 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } }
Aug 06 22:47:05 dist-test-slave-zr1t krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056424, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
I20260806 22:47:05.279556 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.256243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.23.100.188:44515)
0806 22:47:05.256421 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.256426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.256445 (+ 19us) server_negotiation.cc:407] Connection header received
0806 22:47:05.256508 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.256514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.256628 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.256727 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.257912 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.258944 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.261126 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.262132 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.262308 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.268864 (+ 6556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:05.268875 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:05.268883 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:05.268945 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:05.273905 (+ 4960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.275083 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.275092 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.275098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.275300 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.275901 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.275909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.275915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.276426 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:05.276627 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:05.278653 (+ 2026us) server_negotiation.cc:299] Negotiation successful
0806 22:47:05.278990 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 22:47:05.280082 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.255408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:44515 (local address 127.0.0.1:36516)
0806 22:47:05.255852 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:05.255872 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:05.255990 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:05.256929 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:05.256937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:05.257465 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:05.257719 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:05.257733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.259139 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.259145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:05.260649 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:05.260671 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.262632 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.262642 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:05.263319 (+ 677us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.264497 (+ 1178us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:05.264539 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:05.268514 (+ 3975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:05.274156 (+ 5642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:05.274165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:05.274172 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:05.274893 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:05.275465 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:05.275473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:05.275479 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:05.275730 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:05.276709 (+ 979us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:05.276717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:05.276942 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 22:47:05.277139 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 22:47:05.277550 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302}
I20260806 22:47:05.283252 25451 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } } }
I20260806 22:47:05.283713 25451 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:05.284708 25408 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:47:05.284834 25378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f5285a0a620a446d9c946c6b515084ff"
I20260806 22:47:05.285401 25378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:47:05.286669 25306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53: Bootstrap starting.
I20260806 22:47:05.288981 25378 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb8ee3386bb04c76a83bb5f3124088a6 in term 1.
I20260806 22:47:05.290585 25131 leader_election.cc:302] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb8ee3386bb04c76a83bb5f3124088a6, f5285a0a620a446d9c946c6b515084ff; no voters:
I20260806 22:47:05.292088 25421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:47:05.296501 25421 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [term 1 LEADER]: Becoming Leader. State: Replica: eb8ee3386bb04c76a83bb5f3124088a6, State: Running, Role: LEADER
I20260806 22:47:05.301812 25421 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } }
I20260806 22:47:05.306948 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.263221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.23.100.188:44515)
0806 22:47:05.263442 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.263450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.274969 (+ 11519us) server_negotiation.cc:407] Connection header received
0806 22:47:05.275414 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.275422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.275504 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.275573 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.291435 (+ 15862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.292576 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.294564 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.295232 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.295342 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.298928 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:05.298934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:05.298939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:05.298978 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:05.303092 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.304193 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.304199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.304203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.304385 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.304889 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.304896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.304902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.305319 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:05.305633 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:05.306268 (+ 635us) server_negotiation.cc:299] Negotiation successful
0806 22:47:05.306525 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 22:47:05.312323 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.251431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.23.100.190:43117)
0806 22:47:05.251620 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.251625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.265390 (+ 13765us) server_negotiation.cc:407] Connection header received
0806 22:47:05.265828 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.265836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.265963 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.266066 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.276060 (+ 9994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.277090 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.287802 (+ 10712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.288476 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.288580 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.296066 (+ 7486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:05.296075 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:05.296083 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:05.296143 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:05.300941 (+ 4798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.303488 (+ 2547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.303497 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.303503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.303698 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.304697 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.304705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.304711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.307559 (+ 2848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:05.309470 (+ 1911us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:05.311369 (+ 1899us) server_negotiation.cc:299] Negotiation successful
0806 22:47:05.311727 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 22:47:05.314416 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:05.251404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.23.100.189:43801)
0806 22:47:05.251619 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.251626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.264077 (+ 12451us) server_negotiation.cc:407] Connection header received
0806 22:47:05.264378 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.264384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.264506 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.264607 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.272021 (+ 7414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.273173 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.275811 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:05.276464 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:05.276582 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:05.289580 (+ 12998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:05.289587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:05.289591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:05.289632 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:05.295115 (+ 5483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.305415 (+ 10300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.305431 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.305437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.306635 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:05.307686 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:05.307694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:05.307700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:05.310895 (+ 3195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:05.311244 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:05.313560 (+ 2316us) server_negotiation.cc:299] Negotiation successful
0806 22:47:05.313931 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:47:05.318185 25306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53: Neither blocks nor log segments found. Creating new log.
I20260806 22:47:05.319964 25306 log.cc:824] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53: Log is configured to *not* fsync() on all Append() calls
I20260806 22:47:05.321979 25454 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8ee3386bb04c76a83bb5f3124088a6. Latest consensus state: current_term: 1 leader_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } } }
I20260806 22:47:05.322456 25454 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [sys.catalog]: This master's current role is: LEADER
I20260806 22:47:05.324580 25306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53: No bootstrap required, opened a new log
I20260806 22:47:05.340268 25306 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } }
I20260806 22:47:05.340806 25306 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9331eefc6d04313888f7f326279bd53, State: Initialized, Role: FOLLOWER
I20260806 22:47:05.341038 25462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:47:05.341454 25306 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } }
I20260806 22:47:05.348858 25466 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } } }
I20260806 22:47:05.349380 25466 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:05.349268 25462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:47:05.351687 25306 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:47:05.351783 25271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a9331eefc6d04313888f7f326279bd53"
I20260806 22:47:05.352310 25271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [term 0 FOLLOWER]: Advancing to term 1
W20260806 22:47:05.354224 25348 tablet.cc:2369] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:47:05.355917 25271 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb8ee3386bb04c76a83bb5f3124088a6 in term 1.
I20260806 22:47:05.377415 25378 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [term 1 FOLLOWER]: Refusing update from remote peer eb8ee3386bb04c76a83bb5f3124088a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:47:05.378563 25454 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:47:05.388856 25271 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [term 1 FOLLOWER]: Refusing update from remote peer eb8ee3386bb04c76a83bb5f3124088a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:47:05.389895 25454 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:47:05.403187 25466 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8ee3386bb04c76a83bb5f3124088a6. Latest consensus state: current_term: 1 leader_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } } }
I20260806 22:47:05.403537 25466 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:05.408398 25484 mvcc.cc:205] Tried to move back new op lower bound from 7315687118333243392 to 7315687118053109760. Current Snapshot: MvccSnapshot[applied={T|T < 7315687118333243392}]
I20260806 22:47:05.410547 25421 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } } }
I20260806 22:47:05.410892 25421 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [sys.catalog]: This master's current role is: LEADER
I20260806 22:47:05.411759 25471 mvcc.cc:205] Tried to move back new op lower bound from 7315687118333243392 to 7315687118053109760. Current Snapshot: MvccSnapshot[applied={T|T < 7315687118333243392}]
I20260806 22:47:05.416877 25451 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8ee3386bb04c76a83bb5f3124088a6. Latest consensus state: current_term: 1 leader_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } } }
I20260806 22:47:05.417203 25451 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:05.418774 25454 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } } }
I20260806 22:47:05.430207 25454 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6 [sys.catalog]: This master's current role is: LEADER
I20260806 22:47:05.430061 25462 catalog_manager.cc:1347] Generated new cluster ID: 20159cfa238347ccaa08951156c6c912
I20260806 22:47:05.430521 25462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:47:05.446748 25466 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } } }
I20260806 22:47:05.447095 25466 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:05.447054 25503 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 22:47:05.424587 25314 master_runner.cc:186] Error getting master registration for 127.23.100.189:43801: OK, instance_id { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" instance_seqno: 1786056424526451 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 22:47:05.455793 25451 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8ee3386bb04c76a83bb5f3124088a6" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 43117 } } peers { permanent_uuid: "a9331eefc6d04313888f7f326279bd53" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 43801 } } peers { permanent_uuid: "f5285a0a620a446d9c946c6b515084ff" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 44515 } } }
I20260806 22:47:05.458956 25451 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:05.462756 25314 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 22:47:05.465312 25494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 22:47:05.465447 25494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 22:47:05.473749 25462 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:47:05.476830 25462 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:47:05.493471 25462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb8ee3386bb04c76a83bb5f3124088a6: Generated new TSK 0
I20260806 22:47:05.494799 25462 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 22:47:05.600982 25418 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.587216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.23.100.190:43117)
0806 22:47:05.587457 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.587468 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.598324 (+ 10856us) server_negotiation.cc:407] Connection header received
0806 22:47:05.598689 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.598698 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.598838 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.598945 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.600562 (+ 1617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47860: BlockingRecv error: recv got EOF from 127.0.0.1:47860 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 22:47:05.603562 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.587261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.23.100.189:43801)
0806 22:47:05.587659 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.587668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.601763 (+ 14095us) server_negotiation.cc:407] Connection header received
0806 22:47:05.602069 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.602077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.602181 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.602280 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.603273 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55834: BlockingRecv error: recv got EOF from 127.0.0.1:55834 (error 108)
Metrics: {"server-negotiator.queue_time_us":243}
W20260806 22:47:05.605257 25412 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.602703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.23.100.188:44515)
0806 22:47:05.602906 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.602912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.603563 (+ 651us) server_negotiation.cc:407] Connection header received
0806 22:47:05.603829 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.603835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.603916 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.604001 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.604967 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36530: BlockingRecv error: recv got EOF from 127.0.0.1:36530 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 22:47:05.612215 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.610097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.23.100.189:43801)
0806 22:47:05.610309 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.610318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.610824 (+ 506us) server_negotiation.cc:407] Connection header received
0806 22:47:05.611046 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.611051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.611158 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.611266 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.611967 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55848: BlockingRecv error: recv got EOF from 127.0.0.1:55848 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 22:47:05.618209 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.616140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.23.100.189:43801)
0806 22:47:05.616373 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.616382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.616887 (+ 505us) server_negotiation.cc:407] Connection header received
0806 22:47:05.617084 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.617089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.617163 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.617263 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.617978 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55858: BlockingRecv error: recv got EOF from 127.0.0.1:55858 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 22:47:05.633378 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.631172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.23.100.189:43801)
0806 22:47:05.631412 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.631421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.631987 (+ 566us) server_negotiation.cc:407] Connection header received
0806 22:47:05.632204 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.632211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.632291 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.632405 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.633127 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55866: BlockingRecv error: recv got EOF from 127.0.0.1:55866 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 22:47:05.642373 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.640198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.23.100.189:43801)
0806 22:47:05.640471 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.640480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.640985 (+ 505us) server_negotiation.cc:407] Connection header received
0806 22:47:05.641202 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.641207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.641286 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.641399 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.642124 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55870: BlockingRecv error: recv got EOF from 127.0.0.1:55870 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260806 22:47:05.655614 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.653417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.23.100.189:43801)
0806 22:47:05.653669 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.653678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.654253 (+ 575us) server_negotiation.cc:407] Connection header received
0806 22:47:05.654468 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.654474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.654564 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.654674 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.655397 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55872: BlockingRecv error: recv got EOF from 127.0.0.1:55872 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 22:47:05.682883 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.680436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.23.100.189:43801)
0806 22:47:05.680669 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.680679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.681267 (+ 588us) server_negotiation.cc:407] Connection header received
0806 22:47:05.681559 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.681567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.681657 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.681769 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.682559 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55886: BlockingRecv error: recv got EOF from 127.0.0.1:55886 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 22:47:05.720258 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.718023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55896 (local address 127.23.100.189:43801)
0806 22:47:05.718258 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.718268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.718787 (+ 519us) server_negotiation.cc:407] Connection header received
0806 22:47:05.719047 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.719052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.719123 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.719233 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.720027 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55896: BlockingRecv error: recv got EOF from 127.0.0.1:55896 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 22:47:05.790151 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.787862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55906 (local address 127.23.100.189:43801)
0806 22:47:05.788066 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.788075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.788598 (+ 523us) server_negotiation.cc:407] Connection header received
0806 22:47:05.788833 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.788838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.788923 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.789043 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.789911 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55906: BlockingRecv error: recv got EOF from 127.0.0.1:55906 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 22:47:05.927942 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:05.925622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.23.100.189:43801)
0806 22:47:05.925890 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:05.925897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:05.926354 (+ 457us) server_negotiation.cc:407] Connection header received
0806 22:47:05.926544 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:05.926549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:05.926625 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:05.926720 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:05.927709 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55922: BlockingRecv error: recv got EOF from 127.0.0.1:55922 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260806 22:47:06.190397 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:06.188440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.23.100.189:43801)
0806 22:47:06.188626 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:06.188630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:06.188924 (+ 294us) server_negotiation.cc:407] Connection header received
0806 22:47:06.189100 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:06.189105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:06.189186 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:06.189289 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:06.190172 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55934: BlockingRecv error: recv got EOF from 127.0.0.1:55934 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 22:47:06.451609 25503 catalog_manager.cc:1259] Loaded cluster ID: 20159cfa238347ccaa08951156c6c912
I20260806 22:47:06.451821 25503 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53: loading cluster ID for follower catalog manager: success
I20260806 22:47:06.454727 25503 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53: acquiring CA information for follower catalog manager: success
I20260806 22:47:06.456708 25503 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9331eefc6d04313888f7f326279bd53: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 22:47:06.467120 25494 catalog_manager.cc:1259] Loaded cluster ID: 20159cfa238347ccaa08951156c6c912
I20260806 22:47:06.467211 25494 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff: loading cluster ID for follower catalog manager: success
I20260806 22:47:06.469699 25494 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff: acquiring CA information for follower catalog manager: success
I20260806 22:47:06.471424 25494 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5285a0a620a446d9c946c6b515084ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 22:47:06.714962 25598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:06.706655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.23.100.189:43801)
0806 22:47:06.707344 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:06.707351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:06.707716 (+ 365us) server_negotiation.cc:407] Connection header received
0806 22:47:06.707974 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:06.707979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:06.708126 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:06.708235 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:06.708874 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:06.709993 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:06.712104 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:06.712609 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:06.712715 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:06.713464 (+ 749us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:06.714144 (+ 680us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:06.714265 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:06.714462 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 22:47:06.714543 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":349,"threads_started":1}
I20260806 22:47:06.730099 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:06.722933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.23.100.188:44515)
0806 22:47:06.723748 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:06.723753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:06.723772 (+ 19us) server_negotiation.cc:407] Connection header received
0806 22:47:06.723869 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:06.723873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:06.724049 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:06.724200 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:06.724728 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:06.725629 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:06.727542 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:06.727987 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:06.728088 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:06.728725 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:06.729323 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:06.729463 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:06.729671 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 22:47:06.729749 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":596,"threads_started":1}
I20260806 22:47:06.766981 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:06.742059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.23.100.190:43117)
0806 22:47:06.757980 (+ 15921us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:06.757989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:06.758017 (+ 28us) server_negotiation.cc:407] Connection header received
0806 22:47:06.758807 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:06.758813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:06.759042 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:06.759241 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:06.760181 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:06.761323 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:06.763841 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:06.764488 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:06.764644 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:06.765184 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:06.765726 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:06.765857 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:06.766547 (+ 690us) server_negotiation.cc:299] Negotiation successful
0806 22:47:06.766651 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15739,"thread_start_us":16491,"threads_started":1}
I20260806 22:47:06.767046 25598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:06.742514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55958 (local address 127.23.100.189:43801)
0806 22:47:06.742721 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:06.742730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:06.758808 (+ 16078us) server_negotiation.cc:407] Connection header received
0806 22:47:06.759015 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:06.759024 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:06.759138 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:06.759255 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:06.759869 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:06.761088 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:06.763750 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:06.765397 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:06.765552 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:06.766179 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:06.766326 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:06.766408 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:06.766645 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 22:47:06.766722 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 22:47:06.768296 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:06.755909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.23.100.188:44515)
0806 22:47:06.756139 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:06.756147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:06.760177 (+ 4030us) server_negotiation.cc:407] Connection header received
0806 22:47:06.760468 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:06.760474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:06.760573 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:06.760638 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:06.762216 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:06.763342 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:06.765947 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:06.766635 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:06.766782 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:06.767400 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:06.767533 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:06.767617 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:06.767887 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 22:47:06.767977 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 22:47:06.772475 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25106
I20260806 22:47:06.773744 25196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:47:06.917680 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 25106
I20260806 22:47:06.946985 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25216
I20260806 22:47:06.948119 25300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:47:07.060480 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 25216
I20260806 22:47:07.092499 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25314
I20260806 22:47:07.093731 25403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:47:07.203245 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 25314
2026-08-06T22:47:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5399 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 22:47:07 dist-test-slave-zr1t krb5kdc[25624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 22:47:07 dist-test-slave-zr1t krb5kdc[25624](info): set up 2 sockets
Aug 06 22:47:07 dist-test-slave-zr1t krb5kdc[25624](info): commencing operation
krb5kdc: starting...
W20260806 22:47:08.560290 23954 mini_kdc.cc:121] Time spent starting KDC: real 1.306s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/test-user.keytab.
Aug 06 22:47:08 dist-test-slave-zr1t krb5kdc[25624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T22:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T22:47:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.100.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
Entry for principal kudu/127.23.100.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab.
I20260806 22:47:08.928344 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:35933
--webserver_interface=127.23.100.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:35933,127.23.100.189:42471,127.23.100.188:42055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 22:47:09.280807 25644 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 22:47:09.281177 25644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:47:09.281308 25644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:47:09.281437 25644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:47:09.288702 25644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:47:09.288792 25644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:47:09.288901 25644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:47:09.288998 25644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:47:09.289093 25644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:47:09.299008 25644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:35933,127.23.100.189:42471,127.23.100.188:42055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.190.keytab
--principal=kudu/127.23.100.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.190:35933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.100.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:47:09.300354 25644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:47:09.301719 25644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:47:09.313704 25657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:09.313985 25653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:09.313676 25654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:47:09.314603 25644 server_base.cc:1034] running on GCE node
I20260806 22:47:09.315465 25644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:47:09.316882 25644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:47:09.318142 25644 hybrid_clock.cc:648] HybridClock initialized: now 1786056429318070 us; error 40 us; skew 500 ppm
I20260806 22:47:09.318518 25644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:47:09 dist-test-slave-zr1t krb5kdc[25624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056429, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:47:09.321745 25644 init.cc:377] Logged in from keytab as kudu/127.23.100.190@KRBTEST.COM (short username kudu)
I20260806 22:47:09.323235 25644 webserver.cc:460] Webserver started at http://127.23.100.190:45695/ using document root <none> and password file <none>
I20260806 22:47:09.323968 25644 fs_manager.cc:359] Metadata directory not provided
I20260806 22:47:09.324103 25644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:47:09.324443 25644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:47:09.327512 25644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/data/instance:
uuid: "8343900759e24698b740695427ef23aa"
format_stamp: "Formatted at 2026-08-06 22:47:09 on dist-test-slave-zr1t"
server_key: "b00559c1ef719f88507c2193961f27ce"
server_key_iv: "7b5934e4d01c48a5bad3a6f77df37276"
server_key_version: "encryptionkey@0"
I20260806 22:47:09.328332 25644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/wal/instance:
uuid: "8343900759e24698b740695427ef23aa"
format_stamp: "Formatted at 2026-08-06 22:47:09 on dist-test-slave-zr1t"
server_key: "b00559c1ef719f88507c2193961f27ce"
server_key_iv: "7b5934e4d01c48a5bad3a6f77df37276"
server_key_version: "encryptionkey@0"
I20260806 22:47:09.333683 25644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 22:47:09.337082 25669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:47:09.338421 25644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 22:47:09.338622 25644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/wal
uuid: "8343900759e24698b740695427ef23aa"
format_stamp: "Formatted at 2026-08-06 22:47:09 on dist-test-slave-zr1t"
server_key: "b00559c1ef719f88507c2193961f27ce"
server_key_iv: "7b5934e4d01c48a5bad3a6f77df37276"
server_key_version: "encryptionkey@0"
I20260806 22:47:09.338861 25644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:47:09.354938 25644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:47:09.358873 25644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:47:09.359148 25644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:47:09.377921 25644 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.190:35933
I20260806 22:47:09.377942 25742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.190:35933 every 8 connection(s)
I20260806 22:47:09.379290 25644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/data/info.pb
I20260806 22:47:09.382782 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 25644
I20260806 22:47:09.382982 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-0/wal/instance
I20260806 22:47:09.383124 25744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:47:09.383576 23954 external_mini_cluster.cc:1428] Setting key 9a2f73ebc55bb5a27a560bb9bc350de4
I20260806 22:47:09.393206 25744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:47:09 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056429, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:47:09.416852 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.397353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.23.100.190:35933)
0806 22:47:09.399350 (+ 1997us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:09.399355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:09.399406 (+ 51us) server_negotiation.cc:407] Connection header received
0806 22:47:09.401038 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:09.401044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:09.401275 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:09.401461 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:09.402696 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.403877 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.405247 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.405850 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.406051 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:09.409471 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:09.409479 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:09.409491 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:09.409563 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:09.412513 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:09.413229 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:09.413237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:09.413241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:09.413387 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:09.413819 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:09.413824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:09.413827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:09.414299 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:09.414519 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:09.415041 (+ 522us) server_negotiation.cc:299] Negotiation successful
0806 22:47:09.415353 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1786,"spinlock_wait_cycles":79616,"thread_start_us":1549,"threads_started":1}
I20260806 22:47:09.416889 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.397353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:35933 (local address 127.0.0.1:45726)
0806 22:47:09.398350 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.398402 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:09.400666 (+ 2264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:09.401510 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:09.401557 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:09.402135 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:09.402471 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:09.402488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.404048 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.404054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:09.405102 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:09.405114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.406103 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.406124 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:09.406376 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:09.407049 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:09.407071 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:09.409225 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:09.412667 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:09.412680 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:09.412691 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:09.413098 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:09.413506 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:09.413513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:09.413538 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:09.413703 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:09.414440 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:09.414448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:09.414666 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 22:47:09.414907 (+ 241us) client_negotiation.cc:240] Negotiation successful
0806 22:47:09.415477 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":201,"thread_start_us":430,"threads_started":1}
WARNING: no policy specified for kudu/127.23.100.189@KRBTEST.COM; defaulting to no policy
I20260806 22:47:09.419751 25744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
I20260806 22:47:09.420831 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.420421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55508)
0806 22:47:09.420574 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.420682 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.189@KRBTEST.COM" created.
W20260806 22:47:09.421141 25670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
I20260806 22:47:09.422689 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.422352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55522)
0806 22:47:09.422487 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.422554 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 22:47:09.422983 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:42471: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab.
Entry for principal kudu/127.23.100.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab.
I20260806 22:47:09.459019 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.189:42471
--webserver_interface=127.23.100.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab
--principal=kudu/127.23.100.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:35933,127.23.100.189:42471,127.23.100.188:42055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:47:09.472330 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } attempt: 1
I20260806 22:47:09.473781 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.473138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55534)
0806 22:47:09.473418 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.473603 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 22:47:09.474810 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.474451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55538)
0806 22:47:09.474602 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.474679 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 22:47:09.475003 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:42471: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
I20260806 22:47:09.543308 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } attempt: 2
I20260806 22:47:09.544533 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.544028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55554)
0806 22:47:09.544296 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.544391 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 22:47:09.545660 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.545214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55558)
0806 22:47:09.545395 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.545465 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 22:47:09.545851 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:42471: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
I20260806 22:47:09.637104 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } attempt: 3
I20260806 22:47:09.638262 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.637818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55562)
0806 22:47:09.638027 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.638126 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260806 22:47:09.639397 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.639032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55576)
0806 22:47:09.639182 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.639255 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 22:47:09.639583 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:42471: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
I20260806 22:47:09.782828 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } attempt: 4
I20260806 22:47:09.784233 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.783519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55584)
0806 22:47:09.783914 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.784041 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260806 22:47:09.785439 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.784982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55592)
0806 22:47:09.785155 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.785268 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 22:47:09.785699 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.189:42471: Network error: Client connection negotiation failed: client connection to 127.23.100.189:42471: connect: Connection refused (error 111)
W20260806 22:47:09.809805 25755 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 22:47:09.810148 25755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:47:09.810283 25755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:47:09.810415 25755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:47:09.817730 25755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:47:09.817828 25755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:47:09.817920 25755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:47:09.818004 25755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:47:09.818106 25755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:47:09.828006 25755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:35933,127.23.100.189:42471,127.23.100.188:42055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.189.keytab
--principal=kudu/127.23.100.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.189:42471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.100.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:47:09.829337 25755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:47:09.830732 25755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:47:09.843401 25764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:09.843312 25765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:09.843586 25767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:47:09.844208 25755 server_base.cc:1034] running on GCE node
I20260806 22:47:09.844964 25755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:47:09.846415 25755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:47:09.847641 25755 hybrid_clock.cc:648] HybridClock initialized: now 1786056429847569 us; error 38 us; skew 500 ppm
I20260806 22:47:09.847983 25755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:47:09 dist-test-slave-zr1t krb5kdc[25624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056429, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:47:09.854319 25755 init.cc:377] Logged in from keytab as kudu/127.23.100.189@KRBTEST.COM (short username kudu)
I20260806 22:47:09.855887 25755 webserver.cc:460] Webserver started at http://127.23.100.189:37137/ using document root <none> and password file <none>
I20260806 22:47:09.856626 25755 fs_manager.cc:359] Metadata directory not provided
I20260806 22:47:09.856762 25755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:47:09.857100 25755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:47:09.860173 25755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/data/instance:
uuid: "a178401492ca4c8e9f14b61b079ac06e"
format_stamp: "Formatted at 2026-08-06 22:47:09 on dist-test-slave-zr1t"
server_key: "d9e8ab8c90a21848c8e0f302abb7ebbd"
server_key_iv: "1fe310e849d6a738d7950e0fc39d7866"
server_key_version: "encryptionkey@0"
I20260806 22:47:09.861013 25755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/wal/instance:
uuid: "a178401492ca4c8e9f14b61b079ac06e"
format_stamp: "Formatted at 2026-08-06 22:47:09 on dist-test-slave-zr1t"
server_key: "d9e8ab8c90a21848c8e0f302abb7ebbd"
server_key_iv: "1fe310e849d6a738d7950e0fc39d7866"
server_key_version: "encryptionkey@0"
I20260806 22:47:09.866257 25755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 22:47:09.869768 25776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:47:09.871137 25755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 22:47:09.871325 25755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/wal
uuid: "a178401492ca4c8e9f14b61b079ac06e"
format_stamp: "Formatted at 2026-08-06 22:47:09 on dist-test-slave-zr1t"
server_key: "d9e8ab8c90a21848c8e0f302abb7ebbd"
server_key_iv: "1fe310e849d6a738d7950e0fc39d7866"
server_key_version: "encryptionkey@0"
I20260806 22:47:09.871559 25755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:47:09.887677 25755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:47:09.891594 25755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:47:09.891875 25755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:47:09.910959 25755 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.189:42471
I20260806 22:47:09.910991 25845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.189:42471 every 8 connection(s)
I20260806 22:47:09.912266 25755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/data/info.pb
I20260806 22:47:09.913803 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 25755
I20260806 22:47:09.914024 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-1/wal/instance
I20260806 22:47:09.914582 23954 external_mini_cluster.cc:1428] Setting key f3c281a6ba883262e2cad928819dc197
I20260806 22:47:09.916697 25847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:47:09.926719 25847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:47:09 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056429, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:47:09.948693 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.930236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45732 (local address 127.23.100.190:35933)
0806 22:47:09.930818 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:09.930827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:09.931695 (+ 868us) server_negotiation.cc:407] Connection header received
0806 22:47:09.933880 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:09.933886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:09.933988 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:09.934115 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:09.935721 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.936494 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.938154 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.938594 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.938741 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:09.942475 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:09.942484 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:09.942489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:09.942539 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:09.945435 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:09.946274 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:09.946278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:09.946281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:09.946415 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:09.946862 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:09.946866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:09.946868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:09.947168 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:09.947334 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:09.948066 (+ 732us) server_negotiation.cc:299] Negotiation successful
0806 22:47:09.948304 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":283,"threads_started":1}
WARNING: no policy specified for kudu/127.23.100.188@KRBTEST.COM; defaulting to no policy
I20260806 22:47:09.949791 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.930806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:35933 (local address 127.0.0.1:45732)
0806 22:47:09.931535 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.931572 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:09.933596 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:09.934365 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:09.934374 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:09.934915 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:09.935488 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:09.935505 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.936669 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.936673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:09.937995 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:09.938008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.938842 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.938849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:09.939089 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:09.939824 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:09.939848 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:09.942162 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:09.945619 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:09.945629 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:09.945649 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:09.946126 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:09.946533 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:09.946538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:09.946542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:09.946729 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:09.947312 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:09.947321 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:09.947686 (+ 365us) client_negotiation.cc:769] Sending connection context
0806 22:47:09.947961 (+ 275us) client_negotiation.cc:240] Negotiation successful
0806 22:47:09.948522 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":322,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.100.188@KRBTEST.COM" created.
I20260806 22:47:09.951793 25847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:47:09 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056429, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
I20260806 22:47:09.969390 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.952477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55594)
0806 22:47:09.952684 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.952705 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:09.952817 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:09.953897 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:09.953904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:09.954470 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:09.954684 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:09.954698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.956497 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.956504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:09.957967 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:09.957985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.959110 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.959118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:09.959345 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:09.960055 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:09.960082 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:09.962873 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:09.966340 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:09.966347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:09.966354 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:09.966910 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:09.967292 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:09.967299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:09.967305 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:09.967508 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:09.968236 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:09.968243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:09.968409 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 22:47:09.968576 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 22:47:09.968924 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 22:47:09.969434 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.952732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55594 (local address 127.23.100.189:42471)
0806 22:47:09.953349 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:09.953355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:09.953384 (+ 29us) server_negotiation.cc:407] Connection header received
0806 22:47:09.953431 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:09.953437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:09.953725 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:09.953891 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:09.954864 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.956331 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.958118 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:09.958953 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:09.959080 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:09.963101 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:09.963113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:09.963127 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:09.963162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:09.966198 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:09.967048 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:09.967056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:09.967058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:09.967176 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:09.967620 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:09.967624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:09.967626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:09.968093 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:09.968259 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:09.968712 (+ 453us) server_negotiation.cc:299] Negotiation successful
0806 22:47:09.969053 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":467,"thread_start_us":319,"threads_started":1}
I20260806 22:47:09.972216 25847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } has no permanent_uuid. Determining permanent_uuid...
I20260806 22:47:09.973271 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.972842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46594)
0806 22:47:09.973003 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.973129 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 22:47:09.973510 25777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
I20260806 22:47:09.975250 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:09.974861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46608)
0806 22:47:09.975039 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:09.975155 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 22:47:09.975539 25847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:42055: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.100.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab.
Entry for principal kudu/127.23.100.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab.
I20260806 22:47:09.994575 23954 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9rdcRI/build/asan/bin/kudu
/tmp/dist-test-task9rdcRI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.188:42055
--webserver_interface=127.23.100.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab
--principal=kudu/127.23.100.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.100.148:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.100.190:35933,127.23.100.189:42471,127.23.100.188:42055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 22:47:10.024930 25847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } attempt: 1
I20260806 22:47:10.026232 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.025778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46610)
0806 22:47:10.025974 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.026076 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 22:47:10.027348 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.027010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46612)
0806 22:47:10.027174 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.027247 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 22:47:10.027583 25847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:42055: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
I20260806 22:47:10.085014 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } attempt: 5
Aug 06 22:47:10 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056429, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
I20260806 22:47:10.095868 25847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } attempt: 2
I20260806 22:47:10.097885 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.096547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46620)
0806 22:47:10.097481 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.097633 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260806 22:47:10.099181 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.098753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46624)
0806 22:47:10.098930 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.099021 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 22:47:10.099509 25847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:42055: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
I20260806 22:47:10.102411 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.086227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.23.100.189:42471)
0806 22:47:10.086658 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.086670 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.086690 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:47:10.086816 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.086823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.086952 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.087079 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.088544 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.089785 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.091128 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.091604 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.091759 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.095654 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.095660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.095663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.095700 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.099452 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.100240 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.100246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.100250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.100400 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.100823 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.100829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.100832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.101148 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.101323 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.101801 (+ 478us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.102051 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260806 22:47:10.102439 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.085907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:55604)
0806 22:47:10.086170 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.086200 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:10.086369 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:10.087328 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:10.087334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:10.087981 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:10.088311 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.088327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.089975 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.089982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.090942 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.090955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.091784 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.091791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.092038 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.092764 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:10.092811 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:10.095417 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:10.099641 (+ 4224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.099648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.099652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.100093 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.100538 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.100543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.100548 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.100700 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.101284 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:10.101290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:10.101460 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 22:47:10.101682 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 22:47:10.102044 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260806 22:47:10.103641 25744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } has no permanent_uuid. Determining permanent_uuid...
I20260806 22:47:10.104666 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.104258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46630)
0806 22:47:10.104442 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.104535 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 22:47:10.105747 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.105314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46644)
0806 22:47:10.105476 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.105634 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 22:47:10.105998 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:42055: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
I20260806 22:47:10.157246 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } attempt: 1
I20260806 22:47:10.158705 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.158102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46652)
0806 22:47:10.158366 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.158490 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260806 22:47:10.159885 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.159481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46658)
0806 22:47:10.159649 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.159745 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 22:47:10.160135 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:42055: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
I20260806 22:47:10.190779 25847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } attempt: 3
I20260806 22:47:10.192126 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.191549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46662)
0806 22:47:10.191792 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.191923 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 22:47:10.193468 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.192955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46674)
0806 22:47:10.193204 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.193322 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260806 22:47:10.193768 25847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:42055: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
I20260806 22:47:10.228420 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } attempt: 2
I20260806 22:47:10.229861 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.229204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46684)
0806 22:47:10.229479 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.229636 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 22:47:10.231060 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.230659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46686)
0806 22:47:10.230821 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.230914 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 22:47:10.231333 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:42055: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
I20260806 22:47:10.337033 25847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } attempt: 4
I20260806 22:47:10.337584 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } attempt: 3
I20260806 22:47:10.338495 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.337913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46692)
0806 22:47:10.338177 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.338288 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 22:47:10.338755 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.338212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46704)
0806 22:47:10.338421 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.338570 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 22:47:10.339574 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.339240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46720)
0806 22:47:10.339396 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.339475 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 22:47:10.339900 25847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:42055: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
I20260806 22:47:10.339991 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.339549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:46724)
0806 22:47:10.339756 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.339855 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 22:47:10.340165 25744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.100.188:42055: Network error: Client connection negotiation failed: client connection to 127.23.100.188:42055: connect: Connection refused (error 111)
W20260806 22:47:10.358676 25857 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 22:47:10.359019 25857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 22:47:10.359095 25857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 22:47:10.359179 25857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 22:47:10.367278 25857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 22:47:10.367386 25857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 22:47:10.367444 25857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 22:47:10.367501 25857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 22:47:10.367573 25857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 22:47:10.378232 25857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.100.148:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.100.190:35933,127.23.100.189:42471,127.23.100.188:42055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/krb5kdc/kudu_127.23.100.188.keytab
--principal=kudu/127.23.100.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.100.188:42055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.100.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 21:40:31 UTC on ccycloud.quasar-sprmbg.root.comops.site
build id 4571537
ASAN enabled
I20260806 22:47:10.379522 25857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 22:47:10.380908 25857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 22:47:10.393257 25864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:10.393206 25865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 22:47:10.393313 25867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 22:47:10.394381 25857 server_base.cc:1034] running on GCE node
I20260806 22:47:10.395228 25857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 22:47:10.396621 25857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 22:47:10.397881 25857 hybrid_clock.cc:648] HybridClock initialized: now 1786056430397784 us; error 64 us; skew 500 ppm
I20260806 22:47:10.398198 25857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 22:47:10 dist-test-slave-zr1t krb5kdc[25624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056430, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 22:47:10.401428 25857 init.cc:377] Logged in from keytab as kudu/127.23.100.188@KRBTEST.COM (short username kudu)
I20260806 22:47:10.402980 25857 webserver.cc:460] Webserver started at http://127.23.100.188:36621/ using document root <none> and password file <none>
I20260806 22:47:10.403664 25857 fs_manager.cc:359] Metadata directory not provided
I20260806 22:47:10.403755 25857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 22:47:10.404042 25857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 22:47:10.407181 25857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/data/instance:
uuid: "05badef5c1db47889128f0123a1b264b"
format_stamp: "Formatted at 2026-08-06 22:47:10 on dist-test-slave-zr1t"
server_key: "a8e2d047bfcb997e1831f94d67d73893"
server_key_iv: "0bb07405a65987e8cb7895c994b9e583"
server_key_version: "encryptionkey@0"
I20260806 22:47:10.408011 25857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/wal/instance:
uuid: "05badef5c1db47889128f0123a1b264b"
format_stamp: "Formatted at 2026-08-06 22:47:10 on dist-test-slave-zr1t"
server_key: "a8e2d047bfcb997e1831f94d67d73893"
server_key_iv: "0bb07405a65987e8cb7895c994b9e583"
server_key_version: "encryptionkey@0"
I20260806 22:47:10.413491 25857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 22:47:10.417218 25880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 22:47:10.418804 25857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 22:47:10.419018 25857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/data,/tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/wal
uuid: "05badef5c1db47889128f0123a1b264b"
format_stamp: "Formatted at 2026-08-06 22:47:10 on dist-test-slave-zr1t"
server_key: "a8e2d047bfcb997e1831f94d67d73893"
server_key_iv: "0bb07405a65987e8cb7895c994b9e583"
server_key_version: "encryptionkey@0"
I20260806 22:47:10.419250 25857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 22:47:10.446266 25857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 22:47:10.450934 25857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 22:47:10.451169 25857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 22:47:10.470969 25857 rpc_server.cc:307] RPC server started. Bound to: 127.23.100.188:42055
I20260806 22:47:10.471016 25950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.100.188:42055 every 8 connection(s)
I20260806 22:47:10.472458 25857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/data/info.pb
I20260806 22:47:10.476758 25952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 22:47:10.481016 23954 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9rdcRI/build/asan/bin/kudu as pid 25857
I20260806 22:47:10.481244 23954 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9rdcRI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786056046488940-23954-0/minicluster-data/master-2/wal/instance
I20260806 22:47:10.481971 23954 external_mini_cluster.cc:1428] Setting key 82c8fa6d95e1b354321bd3674dfd12b9
I20260806 22:47:10.485726 25952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:47:10 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
Aug 06 22:47:10 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056430, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:47:10.505784 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.489434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.23.100.190:35933)
0806 22:47:10.490045 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.490052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.490563 (+ 511us) server_negotiation.cc:407] Connection header received
0806 22:47:10.490727 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.490731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.490818 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.490925 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.492206 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.493290 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.495519 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.496233 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.496395 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.500008 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.500013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.500016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.500056 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.502846 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.503567 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.503571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.503574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.503699 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.504124 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.504128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.504130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.504383 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.504549 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.505127 (+ 578us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.505362 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":301,"threads_started":1}
I20260806 22:47:10.506886 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.484106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.23.100.188:42055)
0806 22:47:10.484954 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.484966 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.485007 (+ 41us) server_negotiation.cc:407] Connection header received
0806 22:47:10.486708 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.486732 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.487173 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.487579 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.489111 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.491262 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.493046 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.494222 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.494424 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.498942 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.498951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.498964 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.499001 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.502482 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.503244 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.503261 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.503263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.503445 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.503883 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.503889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.503893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.504422 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.504790 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.505155 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.505549 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":400,"threads_started":1}
I20260806 22:47:10.506918 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.489437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.190:35933 (local address 127.0.0.1:35514)
0806 22:47:10.490391 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.490452 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:10.490604 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:10.491010 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:10.491017 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:10.491790 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:10.492036 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.492044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.493470 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.493482 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.495346 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.495360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.496502 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.496508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.496644 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.497326 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:10.497350 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:10.499802 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:10.503004 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.503015 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.503028 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.503431 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.503827 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.503831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.503835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.503997 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.504535 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:10.504543 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:10.504758 (+ 215us) client_negotiation.cc:769] Sending connection context
0806 22:47:10.505015 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 22:47:10.505725 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":219,"thread_start_us":531,"threads_started":1}
I20260806 22:47:10.508770 25952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:47:10 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056430, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
I20260806 22:47:10.517412 25744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } attempt: 4
Aug 06 22:47:10 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.190@KRBTEST.COM
I20260806 22:47:10.523662 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.509772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.23.100.189:42471)
0806 22:47:10.509970 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.509974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.509992 (+ 18us) server_negotiation.cc:407] Connection header received
0806 22:47:10.510089 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.510093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.510152 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.510286 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.511077 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.511754 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.512924 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.513365 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.513552 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.516920 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.516926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.516929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.516965 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.520721 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.521364 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.521371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.521375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.521618 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.522100 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.522108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.522111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.522443 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.522619 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.523097 (+ 478us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.523334 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 22:47:10.523751 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.509591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.189:42471 (local address 127.0.0.1:37666)
0806 22:47:10.509841 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.509863 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:10.509982 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:10.510339 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:10.510346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:10.510784 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:10.510923 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.510931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.511895 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.511902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.512782 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.512794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.513576 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.513582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.513789 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.514407 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:10.514423 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:10.516697 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:10.520831 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.520837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.520841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.521234 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.521752 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.521761 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.521767 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.521963 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.522581 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:10.522586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:10.522752 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 22:47:10.522965 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 22:47:10.523324 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260806 22:47:10.524768 25952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 22:47:10 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056429, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.190@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
I20260806 22:47:10.529903 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.510573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.23.100.190:35933)
0806 22:47:10.510792 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.510797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.510812 (+ 15us) server_negotiation.cc:407] Connection header received
0806 22:47:10.510862 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.510865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.510933 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.511048 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.512285 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.513027 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.514684 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.515164 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.515314 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.520824 (+ 5510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.520832 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.520838 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.520880 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.526708 (+ 5828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.527525 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.527529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.527531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.527679 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.528138 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.528143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.528146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.528465 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.528763 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.529070 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.529299 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 22:47:10 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056430, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.188@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
I20260806 22:47:10.536216 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.518446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:58536)
0806 22:47:10.518732 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.518751 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:10.518857 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:10.519568 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:10.519572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:10.519970 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:10.520138 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.520148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.521543 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.521549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.523181 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.523196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.524546 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.524552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.524773 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.525578 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:10.525611 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:10.528511 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:10.533029 (+ 4518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.533038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.533045 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.533728 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.534263 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.534269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.534272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.534442 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.535155 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:10.535160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:10.535317 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 22:47:10.535500 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 22:47:10.535811 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260806 22:47:10.536342 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.518536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.23.100.188:42055)
0806 22:47:10.519242 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.519248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.519268 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:47:10.519320 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.519326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.519408 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.519516 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.520296 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.521353 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.523334 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.524302 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.524451 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.528765 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.528773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.528788 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.528827 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.532844 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.533940 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.533945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.533949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.534119 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.534599 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.534605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.534608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.534996 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.535171 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.535621 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.535914 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573}
I20260806 22:47:10.542598 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.525555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:58542)
0806 22:47:10.525741 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.525764 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:10.525871 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:10.526847 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:10.526855 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:10.527421 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:10.527615 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.527629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.529441 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.529452 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.531091 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.531109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.532257 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.532263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.532479 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.533133 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:10.533150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:10.535516 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:10.539589 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.539597 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.539601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.540110 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.540537 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.540544 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.540549 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.540705 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.541456 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:10.541463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:10.541670 (+ 207us) client_negotiation.cc:769] Sending connection context
0806 22:47:10.541827 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 22:47:10.542155 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 22:47:10.542690 25744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa: Bootstrap starting.
Aug 06 22:47:10 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.100.189@KRBTEST.COM
I20260806 22:47:10.543531 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.525854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.23.100.188:42055)
0806 22:47:10.526493 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.526499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.526522 (+ 23us) server_negotiation.cc:407] Connection header received
0806 22:47:10.526569 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.526573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.526658 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.526772 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.527808 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.529261 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.531251 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.531920 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.532052 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.535728 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.535737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.535742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.535776 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.539440 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.540247 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.540254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.540257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.540394 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.540824 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.540830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.540833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.541255 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.541409 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.542964 (+ 1555us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.543200 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":322,"threads_started":1}
I20260806 22:47:10.546885 25744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa: Neither blocks nor log segments found. Creating new log.
I20260806 22:47:10.548022 25744 log.cc:824] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa: Log is configured to *not* fsync() on all Append() calls
I20260806 22:47:10.549188 25952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b: Bootstrap starting.
I20260806 22:47:10.551358 25744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa: No bootstrap required, opened a new log
I20260806 22:47:10.551805 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.531865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37680 (local address 127.23.100.189:42471)
0806 22:47:10.532050 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.532056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.532223 (+ 167us) server_negotiation.cc:407] Connection header received
0806 22:47:10.532411 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.532418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.532504 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.532613 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.534457 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.535614 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.537546 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.538204 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.538335 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.544838 (+ 6503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.544848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.544853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.544900 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.548478 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.549464 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.549471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.549477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.549672 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.550160 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.550165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.550168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.550534 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.550803 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.551229 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.551468 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 22:47:10.553282 25952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b: Neither blocks nor log segments found. Creating new log.
I20260806 22:47:10.554595 25952 log.cc:824] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b: Log is configured to *not* fsync() on all Append() calls
I20260806 22:47:10.557228 23954 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 22:47:10.557871 25952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b: No bootstrap required, opened a new log
I20260806 22:47:10.560859 25706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8343900759e24698b740695427ef23aa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35522
I20260806 22:47:10.562705 25744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } }
I20260806 22:47:10.563072 25744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8343900759e24698b740695427ef23aa, State: Initialized, Role: FOLLOWER
I20260806 22:47:10.563546 25744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } }
I20260806 22:47:10.564602 25967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } } }
I20260806 22:47:10.565069 25967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:10.565943 25744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:47:10.566088 25706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 22:47:10.566282 25706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:47:10.567782 25952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } }
I20260806 22:47:10.568192 25952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05badef5c1db47889128f0123a1b264b, State: Initialized, Role: FOLLOWER
I20260806 22:47:10.568682 25952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } }
I20260806 22:47:10.570094 25706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } }
I20260806 22:47:10.569849 25972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } } }
I20260806 22:47:10.570454 25972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:10.570955 25952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:47:10.573232 25922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8343900759e24698b740695427ef23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "05badef5c1db47889128f0123a1b264b"
I20260806 22:47:10.573783 25922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:47:10.578374 25922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8343900759e24698b740695427ef23aa in term 1.
I20260806 22:47:10.579229 25670 leader_election.cc:302] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05badef5c1db47889128f0123a1b264b, 8343900759e24698b740695427ef23aa; no voters:
I20260806 22:47:10.579877 25967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 22:47:10.579965 25706 leader_election.cc:288] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [CANDIDATE]: Term 1 election: Requested vote from peers a178401492ca4c8e9f14b61b079ac06e (127.23.100.189:42471), 05badef5c1db47889128f0123a1b264b (127.23.100.188:42055)
I20260806 22:47:10.584781 25967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [term 1 LEADER]: Becoming Leader. State: Replica: 8343900759e24698b740695427ef23aa, State: Running, Role: LEADER
I20260806 22:47:10.585748 25967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } }
I20260806 22:47:10.590456 25979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8343900759e24698b740695427ef23aa. Latest consensus state: current_term: 1 leader_uuid: "8343900759e24698b740695427ef23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } } }
I20260806 22:47:10.590958 25979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [sys.catalog]: This master's current role is: LEADER
W20260806 22:47:10.612020 25680 tablet.cc:2369] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:47:10.605118 25644 master_runner.cc:186] Error getting master registration for 127.23.100.189:42471: OK, instance_id { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" instance_seqno: 1786056429905689 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 22:47:10.621243 25644 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 22:47:10.624737 26002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 22:47:10.625006 26002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 22:47:10.625658 25992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 22:47:10.628204 25992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 22:47:10.632277 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.584743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.23.100.190:35933)
0806 22:47:10.584922 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.584928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.593987 (+ 9059us) server_negotiation.cc:407] Connection header received
0806 22:47:10.594235 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.594245 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.594379 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.594533 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.597916 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.599532 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.609335 (+ 9803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.610196 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.610296 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.616879 (+ 6583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.616888 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.616893 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.616943 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.621149 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.626561 (+ 5412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.626572 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.626575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.626772 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.629656 (+ 2884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.629662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.629665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.630107 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.630418 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.631557 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.631842 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 22:47:10.635305 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.594823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.23.100.189:42471)
0806 22:47:10.595083 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.595093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.600049 (+ 4956us) server_negotiation.cc:407] Connection header received
0806 22:47:10.601883 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.601893 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.602009 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.602103 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.608210 (+ 6107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.609590 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.617687 (+ 8097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.618445 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.618555 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.624327 (+ 5772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.624336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.624342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.624399 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.628629 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.630341 (+ 1712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.630347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.630352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.630533 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.631382 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.631389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.631393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.631821 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.632084 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.633870 (+ 1786us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.634333 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:47:10.639122 25847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } attempt: 5
I20260806 22:47:10.647284 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.600755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58546 (local address 127.23.100.188:42055)
0806 22:47:10.600996 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.601005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.601849 (+ 844us) server_negotiation.cc:407] Connection header received
0806 22:47:10.602782 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.602788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.602891 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.602970 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.613002 (+ 10032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.614166 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.619530 (+ 5364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.620249 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.620387 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.632641 (+ 12254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.632649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.632654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.632702 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.637402 (+ 4700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.638609 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.638619 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.638625 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.638818 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.642400 (+ 3582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.642408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.642411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.642922 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.643236 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.646419 (+ 3183us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.646768 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
Aug 06 22:47:10 dist-test-slave-zr1t krb5kdc[25624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786056429, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.100.189@KRBTEST.COM for kudu/127.23.100.188@KRBTEST.COM
I20260806 22:47:10.665652 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.640628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.23.100.188:42055)
0806 22:47:10.640834 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.640841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.640861 (+ 20us) server_negotiation.cc:407] Connection header received
0806 22:47:10.640914 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.640922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.641033 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.641146 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.642392 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.643673 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.646034 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.646838 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.646971 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.652993 (+ 6022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 22:47:10.653001 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 22:47:10.653006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 22:47:10.653048 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 22:47:10.658745 (+ 5697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.660216 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.660255 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.660259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.660544 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 22:47:10.661452 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 22:47:10.661483 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 22:47:10.661489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 22:47:10.662238 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 22:47:10.662819 (+ 581us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.663647 (+ 828us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.664274 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 22:47:10.666836 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.639782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.100.188:42055 (local address 127.0.0.1:58552)
0806 22:47:10.640202 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0806 22:47:10.640225 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 22:47:10.640366 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 22:47:10.641190 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 22:47:10.641198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 22:47:10.641915 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0806 22:47:10.642193 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.642208 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.643846 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.643855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.645848 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 22:47:10.645868 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.647591 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.647601 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 22:47:10.647945 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.648711 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 22:47:10.648743 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 22:47:10.652672 (+ 3929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 22:47:10.659069 (+ 6397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.659098 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.659103 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.659997 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.660827 (+ 830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 22:47:10.660832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 22:47:10.660859 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 22:47:10.661200 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 22:47:10.662854 (+ 1654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 22:47:10.662881 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 22:47:10.663142 (+ 261us) client_negotiation.cc:769] Sending connection context
0806 22:47:10.663966 (+ 824us) client_negotiation.cc:240] Negotiation successful
0806 22:47:10.665019 (+ 1053us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260806 22:47:10.679497 25847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e: Bootstrap starting.
I20260806 22:47:10.685101 25847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e: Neither blocks nor log segments found. Creating new log.
I20260806 22:47:10.690567 25922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [term 1 FOLLOWER]: Refusing update from remote peer 8343900759e24698b740695427ef23aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:47:10.691967 25979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:47:10.693001 25847 log.cc:824] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e: Log is configured to *not* fsync() on all Append() calls
W20260806 22:47:10.703261 25889 tablet.cc:2369] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 22:47:10.709787 25847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e: No bootstrap required, opened a new log
I20260806 22:47:10.711829 25972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8343900759e24698b740695427ef23aa. Latest consensus state: current_term: 1 leader_uuid: "8343900759e24698b740695427ef23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } } }
I20260806 22:47:10.712144 25972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:10.718593 26021 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 22:47:10.720009 25979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8343900759e24698b740695427ef23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } } }
I20260806 22:47:10.720305 25979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [sys.catalog]: This master's current role is: LEADER
I20260806 22:47:10.722548 25972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8343900759e24698b740695427ef23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } } }
I20260806 22:47:10.722872 25972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:10.727149 25847 raft_consensus.cc:348] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } }
I20260806 22:47:10.727555 25847 raft_consensus.cc:729] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a178401492ca4c8e9f14b61b079ac06e, State: Initialized, Role: FOLLOWER
I20260806 22:47:10.728127 25847 consensus_queue.cc:260] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } }
I20260806 22:47:10.735221 26024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } } }
I20260806 22:47:10.737636 26024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:10.737464 25847 sys_catalog.cc:564] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 22:47:10.738057 25807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [term 0 FOLLOWER]: Advancing to term 1
I20260806 22:47:10.741770 25810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8343900759e24698b740695427ef23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a178401492ca4c8e9f14b61b079ac06e"
I20260806 22:47:10.742751 25992 catalog_manager.cc:1347] Generated new cluster ID: 96d6aec6465647f3b6c1e65099b771fa
I20260806 22:47:10.742861 25992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 22:47:10.744603 25807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [term 1 FOLLOWER]: Refusing update from remote peer 8343900759e24698b740695427ef23aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 22:47:10.726765 25857 master_runner.cc:186] Error getting master registration for 127.23.100.189:42471: OK, instance_id { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" instance_seqno: 1786056429905689 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 22:47:10.745327 25857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 22:47:10.745719 26018 mvcc.cc:205] Tried to move back new op lower bound from 7315687140007501824 to 7315687139686809600. Current Snapshot: MvccSnapshot[applied={T|T < 7315687140007501824}]
I20260806 22:47:10.748099 25810 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8343900759e24698b740695427ef23aa in term 1.
I20260806 22:47:10.749990 25967 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 22:47:10.760170 25992 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 22:47:10.762125 25992 catalog_manager.cc:1504] Loading token signing keys...
I20260806 22:47:10.779130 25992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa: Generated new TSK 0
I20260806 22:47:10.780622 25992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 22:47:10.786557 26024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8343900759e24698b740695427ef23aa. Latest consensus state: current_term: 1 leader_uuid: "8343900759e24698b740695427ef23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } } }
I20260806 22:47:10.786932 26024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:10.787281 26034 sys_catalog.cc:455] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8343900759e24698b740695427ef23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } } }
I20260806 22:47:10.787612 26034 sys_catalog.cc:458] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 22:47:10.788273 25967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8343900759e24698b740695427ef23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8343900759e24698b740695427ef23aa" member_type: VOTER last_known_addr { host: "127.23.100.190" port: 35933 } } peers { permanent_uuid: "a178401492ca4c8e9f14b61b079ac06e" member_type: VOTER last_known_addr { host: "127.23.100.189" port: 42471 } } peers { permanent_uuid: "05badef5c1db47889128f0123a1b264b" member_type: VOTER last_known_addr { host: "127.23.100.188" port: 42055 } } }
I20260806 22:47:10.788753 25967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8343900759e24698b740695427ef23aa [sys.catalog]: This master's current role is: LEADER
I20260806 22:47:10.819203 26046 catalog_manager.cc:1259] Loaded cluster ID: 96d6aec6465647f3b6c1e65099b771fa
I20260806 22:47:10.819406 26046 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e: loading cluster ID for follower catalog manager: success
I20260806 22:47:10.824772 26046 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e: acquiring CA information for follower catalog manager: success
I20260806 22:47:10.827881 26046 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a178401492ca4c8e9f14b61b079ac06e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 22:47:10.896974 25960 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:10.893649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.23.100.190:35933)
0806 22:47:10.894095 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.894103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.894750 (+ 647us) server_negotiation.cc:407] Connection header received
0806 22:47:10.895104 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.895111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.895208 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.895300 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.896675 (+ 1375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35530: BlockingRecv error: recv got EOF from 127.0.0.1:35530 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 22:47:10.899823 25853 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:10.896515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.23.100.189:42471)
0806 22:47:10.896721 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.896727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.897889 (+ 1162us) server_negotiation.cc:407] Connection header received
0806 22:47:10.898214 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.898221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.898309 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.898444 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.899564 (+ 1120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37704: BlockingRecv error: recv got EOF from 127.0.0.1:37704 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 22:47:10.903491 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:10.901235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58558 (local address 127.23.100.188:42055)
0806 22:47:10.901430 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.901437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.901763 (+ 326us) server_negotiation.cc:407] Connection header received
0806 22:47:10.901924 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.901930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.902020 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.902138 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.903218 (+ 1080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58558: BlockingRecv error: recv got EOF from 127.0.0.1:58558 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 22:47:10.933588 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.911867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.23.100.189:42471)
0806 22:47:10.912073 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.912086 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.924288 (+ 12202us) server_negotiation.cc:407] Connection header received
0806 22:47:10.924689 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.924696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.924915 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.925091 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:10.925991 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.927376 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.929885 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.930719 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.930837 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.931723 (+ 886us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:10.932516 (+ 793us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:10.932691 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.932994 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.933130 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 22:47:10.947937 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:10.945380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.23.100.188:42055)
0806 22:47:10.945615 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.945622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.946208 (+ 586us) server_negotiation.cc:407] Connection header received
0806 22:47:10.946454 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.946460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.946591 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.946715 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.947697 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58564: BlockingRecv error: recv got EOF from 127.0.0.1:58564 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 22:47:10.961272 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.952860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.23.100.189:42471)
0806 22:47:10.953066 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.953074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.954214 (+ 1140us) server_negotiation.cc:407] Connection header received
0806 22:47:10.954492 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.954501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.954598 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.954672 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:10.955408 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.956607 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.958975 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.959703 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.959835 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.960401 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:10.960575 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:10.960667 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.960851 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.960944 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 22:47:10.977689 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:10.969493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.23.100.188:42055)
0806 22:47:10.969714 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.969720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.975628 (+ 5908us) server_negotiation.cc:407] Connection header received
0806 22:47:10.975837 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.975843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.975981 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.976127 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:10.977406 (+ 1279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58566: BlockingRecv error: recv got EOF from 127.0.0.1:58566 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 22:47:10.998376 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:10.987315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.23.100.189:42471)
0806 22:47:10.987486 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:10.987491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:10.990148 (+ 2657us) server_negotiation.cc:407] Connection header received
0806 22:47:10.990355 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:10.990362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:10.990473 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:10.990590 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:10.991098 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.992450 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.994672 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:10.995500 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:10.995646 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:10.996128 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:10.996282 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:10.997794 (+ 1512us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:10.997875 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 22:47:10.997983 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 22:47:11.009040 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:11.006601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.23.100.188:42055)
0806 22:47:11.006798 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.006806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.007349 (+ 543us) server_negotiation.cc:407] Connection header received
0806 22:47:11.007570 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.007576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.007687 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.007805 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:11.008772 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58578: BlockingRecv error: recv got EOF from 127.0.0.1:58578 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 22:47:11.024797 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:11.016952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.23.100.189:42471)
0806 22:47:11.017209 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.017216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.017830 (+ 614us) server_negotiation.cc:407] Connection header received
0806 22:47:11.018053 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.018059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.018162 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.018272 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:11.018813 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.020008 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.022348 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.023153 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.023282 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:11.023831 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:11.023984 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:11.024085 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:11.024369 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 22:47:11.024477 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260806 22:47:11.036160 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:11.033272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.23.100.188:42055)
0806 22:47:11.033585 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.033592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.034223 (+ 631us) server_negotiation.cc:407] Connection header received
0806 22:47:11.034572 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.034582 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.034690 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.034818 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:11.035871 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58580: BlockingRecv error: recv got EOF from 127.0.0.1:58580 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
I20260806 22:47:11.073087 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:11.065067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.23.100.189:42471)
0806 22:47:11.065267 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.065274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.065852 (+ 578us) server_negotiation.cc:407] Connection header received
0806 22:47:11.066089 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.066095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.066202 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.066357 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:11.066912 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.068285 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.070480 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.071353 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.071506 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:11.072044 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:11.072233 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:11.072341 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:11.072596 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 22:47:11.072703 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 22:47:11.084205 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:11.081613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.23.100.188:42055)
0806 22:47:11.081813 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.081819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.082437 (+ 618us) server_negotiation.cc:407] Connection header received
0806 22:47:11.082635 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.082642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.082764 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.082889 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:11.083926 (+ 1037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58590: BlockingRecv error: recv got EOF from 127.0.0.1:58590 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 22:47:11.120448 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:11.112187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.23.100.189:42471)
0806 22:47:11.112417 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.112424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.113084 (+ 660us) server_negotiation.cc:407] Connection header received
0806 22:47:11.113317 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.113323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.113432 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.113590 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:11.114193 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.115527 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.117953 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.118668 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.118828 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:11.119471 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:11.119670 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:11.119778 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:11.120009 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 22:47:11.120124 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 22:47:11.131572 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:11.128852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.23.100.188:42055)
0806 22:47:11.129095 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.129103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.129772 (+ 669us) server_negotiation.cc:407] Connection header received
0806 22:47:11.130013 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.130019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.130137 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.130269 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:11.131315 (+ 1046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58600: BlockingRecv error: recv got EOF from 127.0.0.1:58600 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 22:47:11.176640 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:11.168149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.23.100.189:42471)
0806 22:47:11.168334 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.168341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.168961 (+ 620us) server_negotiation.cc:407] Connection header received
0806 22:47:11.169167 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.169175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.169283 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.169408 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:11.170009 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.171335 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.173920 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.174783 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.174932 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:11.175644 (+ 712us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:11.175806 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:11.175909 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:11.176168 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 22:47:11.176270 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 22:47:11.188213 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:11.185569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.23.100.188:42055)
0806 22:47:11.185784 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.185791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.186433 (+ 642us) server_negotiation.cc:407] Connection header received
0806 22:47:11.186658 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.186664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.186768 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.186880 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:11.187928 (+ 1048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58602: BlockingRecv error: recv got EOF from 127.0.0.1:58602 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 22:47:11.263839 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:11.256980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.23.100.189:42471)
0806 22:47:11.257251 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.257259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.257601 (+ 342us) server_negotiation.cc:407] Connection header received
0806 22:47:11.257856 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.257862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.257985 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.258109 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:11.258617 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.259992 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.261441 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.262262 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.262417 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:11.262902 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:11.263075 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:11.263197 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:11.263394 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 22:47:11.263504 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260806 22:47:11.274602 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:11.271726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.23.100.188:42055)
0806 22:47:11.271922 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.271926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.272674 (+ 748us) server_negotiation.cc:407] Connection header received
0806 22:47:11.272937 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.272941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.273032 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.273118 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:11.274311 (+ 1193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58612: BlockingRecv error: recv got EOF from 127.0.0.1:58612 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 22:47:11.412921 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:11.407083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.23.100.189:42471)
0806 22:47:11.407368 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.407375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.407723 (+ 348us) server_negotiation.cc:407] Connection header received
0806 22:47:11.408002 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.408007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.408080 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.408189 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:11.408657 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.409353 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.411071 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.411553 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.411705 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:11.412138 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:11.412257 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:11.412360 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:11.412520 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 22:47:11.412606 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260806 22:47:11.421228 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:11.419138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.23.100.188:42055)
0806 22:47:11.419323 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.419328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.419799 (+ 471us) server_negotiation.cc:407] Connection header received
0806 22:47:11.419956 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.419960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.420038 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.420124 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:11.420951 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58624: BlockingRecv error: recv got EOF from 127.0.0.1:58624 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 22:47:11.688184 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:11.683119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.23.100.189:42471)
0806 22:47:11.683401 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.683407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.683619 (+ 212us) server_negotiation.cc:407] Connection header received
0806 22:47:11.683795 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.683799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.683870 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.683959 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:11.684455 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.685175 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.686589 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:11.686980 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:11.687075 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:11.687499 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:11.687601 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:11.687668 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:11.687816 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 22:47:11.687882 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260806 22:47:11.696395 25959 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 22:47:11.694314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.23.100.188:42055)
0806 22:47:11.694608 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:11.694617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:11.694864 (+ 247us) server_negotiation.cc:407] Connection header received
0806 22:47:11.695088 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:11.695094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:11.695186 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:11.695303 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 22:47:11.696162 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58628: BlockingRecv error: recv got EOF from 127.0.0.1:58628 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20260806 22:47:11.723783 26021 catalog_manager.cc:1259] Loaded cluster ID: 96d6aec6465647f3b6c1e65099b771fa
I20260806 22:47:11.723944 26021 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b: loading cluster ID for follower catalog manager: success
I20260806 22:47:11.726676 26021 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b: acquiring CA information for follower catalog manager: success
I20260806 22:47:11.728659 26021 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 05badef5c1db47889128f0123a1b264b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 22:47:12.217756 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:12.212106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.23.100.189:42471)
0806 22:47:12.212662 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:12.212667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:12.212736 (+ 69us) server_negotiation.cc:407] Connection header received
0806 22:47:12.212925 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:12.212929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:12.213011 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:12.213106 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:12.213558 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:12.214326 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:12.215786 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:12.216230 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:12.216348 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:12.216836 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:12.216947 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:12.217024 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:12.217255 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 22:47:12.217372 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":308,"threads_started":1}
I20260806 22:47:12.235778 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:12.229216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.23.100.188:42055)
0806 22:47:12.229779 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:12.229786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:12.229810 (+ 24us) server_negotiation.cc:407] Connection header received
0806 22:47:12.229911 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:12.229917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:12.230173 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:12.230358 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:12.230853 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:12.231655 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:12.233143 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:12.233696 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:12.233846 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:12.234441 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:12.235047 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:12.235236 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:12.235311 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 22:47:12.235412 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":242,"threads_started":1}
I20260806 22:47:12.250852 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:12.244036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.23.100.189:42471)
0806 22:47:12.244239 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:12.244245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:12.244585 (+ 340us) server_negotiation.cc:407] Connection header received
0806 22:47:12.244888 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:12.244897 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:12.245032 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:12.245154 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:12.246229 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:12.247157 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:12.248769 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:12.249418 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:12.249620 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:12.250045 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:12.250162 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:12.250270 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:12.250433 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 22:47:12.250521 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 22:47:12.253109 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:12.242908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.23.100.190:35933)
0806 22:47:12.243397 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:12.243404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:12.244143 (+ 739us) server_negotiation.cc:407] Connection header received
0806 22:47:12.244302 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:12.244309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:12.244551 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:12.245106 (+ 555us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:12.245694 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:12.246653 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:12.249663 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:12.250572 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:12.250724 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:12.251675 (+ 951us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:12.252241 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:12.252383 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:12.252554 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 22:47:12.252680 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":294,"threads_started":1}
I20260806 22:47:12.256615 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 22:47:12.244595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.23.100.188:42055)
0806 22:47:12.244785 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 22:47:12.244790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 22:47:12.250846 (+ 6056us) server_negotiation.cc:407] Connection header received
0806 22:47:12.251140 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 22:47:12.251147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 22:47:12.251275 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 22:47:12.251347 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 22:47:12.251865 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:12.252795 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:12.254602 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 22:47:12.255224 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 22:47:12.255323 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 22:47:12.255837 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 22:47:12.255969 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 22:47:12.256076 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 22:47:12.256237 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 22:47:12.256321 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 22:47:12.259320 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25644
I20260806 22:47:12.260918 25732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:47:12.373641 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 25644
I20260806 22:47:12.404546 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25755
I20260806 22:47:12.405843 25840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:47:12.521785 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 25755
I20260806 22:47:12.548890 23954 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9rdcRI/build/asan/bin/kudu pid 25857
I20260806 22:47:12.550203 25946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 22:47:12.658383 23954 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9rdcRI/build/asan/bin/kudu with pid 25857
2026-08-06T22:47:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5454 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10853 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (386141 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST